[==========] Running 56 tests from 4 test suites.
[----------] Global test environment set-up.
[----------] 30 tests from SecurityITest
[ RUN ] SecurityITest.TestAuthorizationOnListTablets
Loading random data
Initializing database '/tmp/dist-test-task1JRg7N/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781955746769051-7083-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 20 11:42:26 dist-test-slave-cjc6 krb5kdc[7090](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 20 11:42:26 dist-test-slave-cjc6 krb5kdc[7090](info): set up 2 sockets
Jun 20 11:42:26 dist-test-slave-cjc6 krb5kdc[7090](info): commencing operation
krb5kdc: starting...
WARNING: Logging before InitGoogleLogging() is written to STDERR
W20260620 11:42:27.822228 7083 mini_kdc.cc:121] Time spent starting KDC: real 1.036s user 0.005s sys 0.000s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task1JRg7N/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781955746769051-7083-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task1JRg7N/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781955746769051-7083-0/krb5kdc/test-user.keytab.
Jun 20 11:42:27 dist-test-slave-cjc6 krb5kdc[7090](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781955747, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-20T11:42:27Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-20T11:42:27Z Disabled control of system clock
WARNING: no policy specified for kudu/127.6.234.254@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.6.234.254@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.6.234.254 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task1JRg7N/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781955746769051-7083-0/krb5kdc/kudu_127.6.234.254.keytab.
Entry for principal kudu/127.6.234.254 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task1JRg7N/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781955746769051-7083-0/krb5kdc/kudu_127.6.234.254.keytab.
I20260620 11:42:27.937914 7083 external_mini_cluster.cc:1326] Running /tmp/dist-test-task1JRg7N/build/release/bin/kudu
/tmp/dist-test-task1JRg7N/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task1JRg7N/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781955746769051-7083-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task1JRg7N/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781955746769051-7083-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task1JRg7N/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781955746769051-7083-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task1JRg7N/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781955746769051-7083-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.6.234.254:38789
--webserver_interface=127.6.234.254
--webserver_port=0
--keytab_file=/tmp/dist-test-task1JRg7N/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781955746769051-7083-0/krb5kdc/kudu_127.6.234.254.keytab
--principal=kudu/127.6.234.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.6.234.212:41303
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.6.234.254:38789
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task1JRg7N/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781955746769051-7083-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task1JRg7N/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781955746769051-7083-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task1JRg7N/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781955746769051-7083-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 11:42:28.017022 7103 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 11:42:28.017247 7103 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 11:42:28.017282 7103 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 11:42:28.018462 7103 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 11:42:28.018528 7103 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 11:42:28.018553 7103 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 11:42:28.018581 7103 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 11:42:28.018605 7103 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 11:42:28.020062 7103 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.6.234.212:41303
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task1JRg7N/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781955746769051-7083-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task1JRg7N/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781955746769051-7083-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.6.234.254:38789
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task1JRg7N/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781955746769051-7083-0/krb5kdc/kudu_127.6.234.254.keytab
--principal=kudu/127.6.234.254
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.6.234.254:38789
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task1JRg7N/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781955746769051-7083-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.6.234.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.7103
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task1JRg7N/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781955746769051-7083-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 20 Jun 2026 10:32:37 UTC on ccycloud.quasar-bgyofv.root.comops.site
build id 4457173
I20260620 11:42:28.020328 7103 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 11:42:28.020552 7103 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 11:42:28.025246 7111 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 11:42:28.025297 7103 server_base.cc:1034] running on GCE node
W20260620 11:42:28.027061 7109 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 11:42:28.027184 7108 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 11:42:28.027521 7103 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 11:42:28.027904 7103 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 11:42:28.029119 7103 hybrid_clock.cc:648] HybridClock initialized: now 1781955748029094 us; error 40 us; skew 500 ppm
Jun 20 11:42:28 dist-test-slave-cjc6 krb5kdc[7090](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781955748, etypes {rep=17 tkt=17 ses=17}, kudu/127.6.234.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 11:42:28.031620 7103 init.cc:377] Logged in from keytab as kudu/127.6.234.254@KRBTEST.COM (short username kudu)
I20260620 11:42:28.032164 7103 webserver.cc:460] Webserver started at http://127.6.234.254:37925/ using document root <none> and password file <none>
I20260620 11:42:28.032418 7103 fs_manager.cc:359] Metadata directory not provided
I20260620 11:42:28.032483 7103 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 11:42:28.032626 7103 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 11:42:28.033664 7103 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task1JRg7N/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781955746769051-7083-0/minicluster-data/master-0/data/instance:
uuid: "a72eff3142634be2bc11c3e90bb50c48"
format_stamp: "Formatted at 2026-06-20 11:42:28 on dist-test-slave-cjc6"
server_key: "1799fcdb88ab83e61f44133b85ebe3b0"
server_key_iv: "ef8a9ddb802bc65aaae0ee09799053fe"
server_key_version: "encryptionkey@0"
I20260620 11:42:28.034085 7103 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task1JRg7N/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781955746769051-7083-0/minicluster-data/master-0/wal/instance:
uuid: "a72eff3142634be2bc11c3e90bb50c48"
format_stamp: "Formatted at 2026-06-20 11:42:28 on dist-test-slave-cjc6"
server_key: "1799fcdb88ab83e61f44133b85ebe3b0"
server_key_iv: "ef8a9ddb802bc65aaae0ee09799053fe"
server_key_version: "encryptionkey@0"
I20260620 11:42:28.035496 7103 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.002s
I20260620 11:42:28.036262 7118 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 11:42:28.036432 7103 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.001s
I20260620 11:42:28.036518 7103 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task1JRg7N/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781955746769051-7083-0/minicluster-data/master-0/data,/tmp/dist-test-task1JRg7N/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781955746769051-7083-0/minicluster-data/master-0/wal
uuid: "a72eff3142634be2bc11c3e90bb50c48"
format_stamp: "Formatted at 2026-06-20 11:42:28 on dist-test-slave-cjc6"
server_key: "1799fcdb88ab83e61f44133b85ebe3b0"
server_key_iv: "ef8a9ddb802bc65aaae0ee09799053fe"
server_key_version: "encryptionkey@0"
I20260620 11:42:28.036602 7103 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task1JRg7N/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781955746769051-7083-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task1JRg7N/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781955746769051-7083-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task1JRg7N/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781955746769051-7083-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 11:42:28.055522 7103 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 11:42:28.057279 7103 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 11:42:28.057437 7103 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 11:42:28.061458 7103 rpc_server.cc:307] RPC server started. Bound to: 127.6.234.254:38789
I20260620 11:42:28.061492 7170 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.6.234.254:38789 every 8 connection(s)
I20260620 11:42:28.061832 7103 server_base.cc:1166] Dumped server information to /tmp/dist-test-task1JRg7N/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781955746769051-7083-0/minicluster-data/master-0/data/info.pb
I20260620 11:42:28.062364 7083 external_mini_cluster.cc:1388] Started /tmp/dist-test-task1JRg7N/build/release/bin/kudu as pid 7103
I20260620 11:42:28.062487 7083 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task1JRg7N/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781955746769051-7083-0/minicluster-data/master-0/wal/instance
I20260620 11:42:28.062752 7171 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 11:42:28.063144 7083 external_mini_cluster.cc:1428] Setting key 3db3d6f1a281a9cc356e3911afc1c99a
I20260620 11:42:28.065546 7171 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P a72eff3142634be2bc11c3e90bb50c48: Bootstrap starting.
I20260620 11:42:28.066252 7171 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P a72eff3142634be2bc11c3e90bb50c48: Neither blocks nor log segments found. Creating new log.
I20260620 11:42:28.066591 7171 log.cc:824] T 00000000000000000000000000000000 P a72eff3142634be2bc11c3e90bb50c48: Log is configured to *not* fsync() on all Append() calls
I20260620 11:42:28.067303 7171 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P a72eff3142634be2bc11c3e90bb50c48: No bootstrap required, opened a new log
I20260620 11:42:28.069049 7171 raft_consensus.cc:348] T 00000000000000000000000000000000 P a72eff3142634be2bc11c3e90bb50c48 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a72eff3142634be2bc11c3e90bb50c48" member_type: VOTER last_known_addr { host: "127.6.234.254" port: 38789 } }
I20260620 11:42:28.069178 7171 raft_consensus.cc:374] T 00000000000000000000000000000000 P a72eff3142634be2bc11c3e90bb50c48 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 11:42:28.069201 7171 raft_consensus.cc:729] T 00000000000000000000000000000000 P a72eff3142634be2bc11c3e90bb50c48 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a72eff3142634be2bc11c3e90bb50c48, State: Initialized, Role: FOLLOWER
I20260620 11:42:28.069334 7171 consensus_queue.cc:260] T 00000000000000000000000000000000 P a72eff3142634be2bc11c3e90bb50c48 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a72eff3142634be2bc11c3e90bb50c48" member_type: VOTER last_known_addr { host: "127.6.234.254" port: 38789 } }
I20260620 11:42:28.069409 7171 raft_consensus.cc:388] T 00000000000000000000000000000000 P a72eff3142634be2bc11c3e90bb50c48 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260620 11:42:28.069430 7171 raft_consensus.cc:482] T 00000000000000000000000000000000 P a72eff3142634be2bc11c3e90bb50c48 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260620 11:42:28.069502 7171 raft_consensus.cc:3037] T 00000000000000000000000000000000 P a72eff3142634be2bc11c3e90bb50c48 [term 0 FOLLOWER]: Advancing to term 1
I20260620 11:42:28.070106 7171 raft_consensus.cc:504] T 00000000000000000000000000000000 P a72eff3142634be2bc11c3e90bb50c48 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a72eff3142634be2bc11c3e90bb50c48" member_type: VOTER last_known_addr { host: "127.6.234.254" port: 38789 } }
I20260620 11:42:28.070228 7171 leader_election.cc:302] T 00000000000000000000000000000000 P a72eff3142634be2bc11c3e90bb50c48 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: a72eff3142634be2bc11c3e90bb50c48; no voters:
I20260620 11:42:28.070405 7171 leader_election.cc:288] T 00000000000000000000000000000000 P a72eff3142634be2bc11c3e90bb50c48 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260620 11:42:28.070446 7176 raft_consensus.cc:2781] T 00000000000000000000000000000000 P a72eff3142634be2bc11c3e90bb50c48 [term 1 FOLLOWER]: Leader election won for term 1
I20260620 11:42:28.070560 7176 raft_consensus.cc:686] T 00000000000000000000000000000000 P a72eff3142634be2bc11c3e90bb50c48 [term 1 LEADER]: Becoming Leader. State: Replica: a72eff3142634be2bc11c3e90bb50c48, State: Running, Role: LEADER
I20260620 11:42:28.070641 7176 consensus_queue.cc:237] T 00000000000000000000000000000000 P a72eff3142634be2bc11c3e90bb50c48 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a72eff3142634be2bc11c3e90bb50c48" member_type: VOTER last_known_addr { host: "127.6.234.254" port: 38789 } }
I20260620 11:42:28.070691 7171 sys_catalog.cc:564] T 00000000000000000000000000000000 P a72eff3142634be2bc11c3e90bb50c48 [sys.catalog]: configured and running, proceeding with master startup.
I20260620 11:42:28.070988 7176 sys_catalog.cc:455] T 00000000000000000000000000000000 P a72eff3142634be2bc11c3e90bb50c48 [sys.catalog]: SysCatalogTable state changed. Reason: New leader a72eff3142634be2bc11c3e90bb50c48. Latest consensus state: current_term: 1 leader_uuid: "a72eff3142634be2bc11c3e90bb50c48" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a72eff3142634be2bc11c3e90bb50c48" member_type: VOTER last_known_addr { host: "127.6.234.254" port: 38789 } } }
I20260620 11:42:28.071092 7176 sys_catalog.cc:458] T 00000000000000000000000000000000 P a72eff3142634be2bc11c3e90bb50c48 [sys.catalog]: This master's current role is: LEADER
I20260620 11:42:28.071002 7178 sys_catalog.cc:455] T 00000000000000000000000000000000 P a72eff3142634be2bc11c3e90bb50c48 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "a72eff3142634be2bc11c3e90bb50c48" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a72eff3142634be2bc11c3e90bb50c48" member_type: VOTER last_known_addr { host: "127.6.234.254" port: 38789 } } }
I20260620 11:42:28.071245 7178 sys_catalog.cc:458] T 00000000000000000000000000000000 P a72eff3142634be2bc11c3e90bb50c48 [sys.catalog]: This master's current role is: LEADER
W20260620 11:42:28.072461 7191 catalog_manager.cc:1558] T 00000000000000000000000000000000 P a72eff3142634be2bc11c3e90bb50c48: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260620 11:42:28.072522 7191 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260620 11:42:28.072650 7183 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260620 11:42:28.072845 7183 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260620 11:42:28.074409 7183 catalog_manager.cc:1347] Generated new cluster ID: a5890fd320324d249cbcfc40f3edfe81
I20260620 11:42:28.074460 7183 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
Jun 20 11:42:28 dist-test-slave-cjc6 krb5kdc[7090](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781955747, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.6.234.254@KRBTEST.COM
I20260620 11:42:28.080559 7174 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 11:42:28.063979 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37964 (local address 127.6.234.254:38789)
0620 11:42:28.064157 (+ 178us) server_negotiation.cc:206] Beginning negotiation
0620 11:42:28.064166 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0620 11:42:28.064381 (+ 215us) server_negotiation.cc:407] Connection header received
0620 11:42:28.067260 (+ 2879us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 11:42:28.067267 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 11:42:28.067312 (+ 45us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 11:42:28.067502 (+ 190us) server_negotiation.cc:226] Negotiated authn=SASL
0620 11:42:28.068865 (+ 1363us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 11:42:28.070031 (+ 1166us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 11:42:28.071486 (+ 1455us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 11:42:28.071887 (+ 401us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 11:42:28.072005 (+ 118us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 11:42:28.076274 (+ 4269us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 11:42:28.076279 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 11:42:28.076279 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 11:42:28.076291 (+ 12us) server_negotiation.cc:922] calling sasl_server_start()
0620 11:42:28.078340 (+ 2049us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 11:42:28.078809 (+ 469us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 11:42:28.078813 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 11:42:28.078814 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 11:42:28.078961 (+ 147us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 11:42:28.079296 (+ 335us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 11:42:28.079297 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 11:42:28.079297 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 11:42:28.079602 (+ 305us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 11:42:28.079766 (+ 164us) server_negotiation.cc:1035] Waiting for connection context
0620 11:42:28.080049 (+ 283us) server_negotiation.cc:299] Negotiation successful
0620 11:42:28.080187 (+ 138us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":9,"server-negotiator.queue_time_us":125,"thread_start_us":59,"threads_started":1}
I20260620 11:42:28.088953 7183 catalog_manager.cc:1370] Generated new certificate authority record
I20260620 11:42:28.089717 7183 catalog_manager.cc:1504] Loading token signing keys...
I20260620 11:42:28.092676 7183 catalog_manager.cc:5868] T 00000000000000000000000000000000 P a72eff3142634be2bc11c3e90bb50c48: Generated new TSK 0
I20260620 11:42:28.092842 7183 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.6.234.193@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.6.234.193@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.6.234.193 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task1JRg7N/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781955746769051-7083-0/krb5kdc/kudu_127.6.234.193.keytab.
Entry for principal kudu/127.6.234.193 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task1JRg7N/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781955746769051-7083-0/krb5kdc/kudu_127.6.234.193.keytab.
I20260620 11:42:28.118325 7083 external_mini_cluster.cc:1326] Running /tmp/dist-test-task1JRg7N/build/release/bin/kudu
/tmp/dist-test-task1JRg7N/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task1JRg7N/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781955746769051-7083-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-task1JRg7N/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781955746769051-7083-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task1JRg7N/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781955746769051-7083-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-task1JRg7N/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781955746769051-7083-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.6.234.193:0
--local_ip_for_outbound_sockets=127.6.234.193
--webserver_interface=127.6.234.193
--webserver_port=0
--tserver_master_addrs=127.6.234.254:38789
--keytab_file=/tmp/dist-test-task1JRg7N/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781955746769051-7083-0/krb5kdc/kudu_127.6.234.193.keytab
--principal=kudu/127.6.234.193
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.6.234.212:41303
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_enforce_access_control with env {KRB5CCNAME=/tmp/dist-test-task1JRg7N/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781955746769051-7083-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task1JRg7N/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781955746769051-7083-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task1JRg7N/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781955746769051-7083-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 11:42:28.198246 7197 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 11:42:28.198472 7197 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 11:42:28.198513 7197 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 11:42:28.199682 7197 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 11:42:28.199749 7197 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 11:42:28.199795 7197 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.6.234.193
I20260620 11:42:28.201226 7197 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.6.234.212:41303
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task1JRg7N/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781955746769051-7083-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-task1JRg7N/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781955746769051-7083-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task1JRg7N/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781955746769051-7083-0/krb5kdc/kudu_127.6.234.193.keytab
--principal=kudu/127.6.234.193
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.6.234.193:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task1JRg7N/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781955746769051-7083-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.6.234.193
--webserver_port=0
--tserver_master_addrs=127.6.234.254:38789
--tserver_enforce_access_control=true
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.7197
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.6.234.193
--log_dir=/tmp/dist-test-task1JRg7N/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781955746769051-7083-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 20 Jun 2026 10:32:37 UTC on ccycloud.quasar-bgyofv.root.comops.site
build id 4457173
I20260620 11:42:28.201493 7197 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 11:42:28.201781 7197 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 11:42:28.206681 7203 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 11:42:28.206759 7202 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 11:42:28.206696 7205 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 11:42:28.206792 7197 server_base.cc:1034] running on GCE node
I20260620 11:42:28.207149 7197 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 11:42:28.207345 7197 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 11:42:28.208492 7197 hybrid_clock.cc:648] HybridClock initialized: now 1781955748208482 us; error 27 us; skew 500 ppm
Jun 20 11:42:28 dist-test-slave-cjc6 krb5kdc[7090](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781955748, etypes {rep=17 tkt=17 ses=17}, kudu/127.6.234.193@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 11:42:28.211031 7197 init.cc:377] Logged in from keytab as kudu/127.6.234.193@KRBTEST.COM (short username kudu)
I20260620 11:42:28.211512 7197 webserver.cc:460] Webserver started at http://127.6.234.193:45383/ using document root <none> and password file <none>
I20260620 11:42:28.211757 7197 fs_manager.cc:359] Metadata directory not provided
I20260620 11:42:28.211827 7197 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 11:42:28.211975 7197 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 11:42:28.213014 7197 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task1JRg7N/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781955746769051-7083-0/minicluster-data/ts-0/data/instance:
uuid: "1a68e21b73f6481b87c885c10e245673"
format_stamp: "Formatted at 2026-06-20 11:42:28 on dist-test-slave-cjc6"
server_key: "844d1929188292a87f3b95c281114bbb"
server_key_iv: "207a03b164a3471a2449f90b1584a176"
server_key_version: "encryptionkey@0"
I20260620 11:42:28.213397 7197 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task1JRg7N/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781955746769051-7083-0/minicluster-data/ts-0/wal/instance:
uuid: "1a68e21b73f6481b87c885c10e245673"
format_stamp: "Formatted at 2026-06-20 11:42:28 on dist-test-slave-cjc6"
server_key: "844d1929188292a87f3b95c281114bbb"
server_key_iv: "207a03b164a3471a2449f90b1584a176"
server_key_version: "encryptionkey@0"
I20260620 11:42:28.214780 7197 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.003s
I20260620 11:42:28.215487 7212 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 11:42:28.215651 7197 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.000s
I20260620 11:42:28.215754 7197 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task1JRg7N/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781955746769051-7083-0/minicluster-data/ts-0/data,/tmp/dist-test-task1JRg7N/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781955746769051-7083-0/minicluster-data/ts-0/wal
uuid: "1a68e21b73f6481b87c885c10e245673"
format_stamp: "Formatted at 2026-06-20 11:42:28 on dist-test-slave-cjc6"
server_key: "844d1929188292a87f3b95c281114bbb"
server_key_iv: "207a03b164a3471a2449f90b1584a176"
server_key_version: "encryptionkey@0"
I20260620 11:42:28.215824 7197 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task1JRg7N/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781955746769051-7083-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-task1JRg7N/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781955746769051-7083-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-task1JRg7N/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781955746769051-7083-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 11:42:28.234895 7197 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 11:42:28.236769 7197 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 11:42:28.236935 7197 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 11:42:28.237157 7197 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 11:42:28.237452 7197 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 11:42:28.237504 7197 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 11:42:28.237545 7197 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 11:42:28.237571 7197 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 11:42:28.243637 7197 rpc_server.cc:307] RPC server started. Bound to: 127.6.234.193:44701
I20260620 11:42:28.243708 7325 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.6.234.193:44701 every 8 connection(s)
I20260620 11:42:28.244022 7197 server_base.cc:1166] Dumped server information to /tmp/dist-test-task1JRg7N/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781955746769051-7083-0/minicluster-data/ts-0/data/info.pb
Jun 20 11:42:28 dist-test-slave-cjc6 krb5kdc[7090](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781955748, etypes {rep=17 tkt=17 ses=17}, kudu/127.6.234.193@KRBTEST.COM for kudu/127.6.234.254@KRBTEST.COM
I20260620 11:42:28.252498 7174 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 11:42:28.244775 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.6.234.193:55761 (local address 127.6.234.254:38789)
0620 11:42:28.244856 (+ 81us) server_negotiation.cc:206] Beginning negotiation
0620 11:42:28.244859 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0620 11:42:28.245023 (+ 164us) server_negotiation.cc:407] Connection header received
0620 11:42:28.245476 (+ 453us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 11:42:28.245477 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 11:42:28.245493 (+ 16us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 11:42:28.245545 (+ 52us) server_negotiation.cc:226] Negotiated authn=SASL
0620 11:42:28.246526 (+ 981us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 11:42:28.247022 (+ 496us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 11:42:28.247953 (+ 931us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 11:42:28.248147 (+ 194us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 11:42:28.248196 (+ 49us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 11:42:28.250268 (+ 2072us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 11:42:28.250269 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 11:42:28.250269 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 11:42:28.250275 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0620 11:42:28.251415 (+ 1140us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 11:42:28.251762 (+ 347us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 11:42:28.251763 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 11:42:28.251763 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 11:42:28.251809 (+ 46us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 11:42:28.252010 (+ 201us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 11:42:28.252011 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 11:42:28.252011 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 11:42:28.252106 (+ 95us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 11:42:28.252153 (+ 47us) server_negotiation.cc:1035] Waiting for connection context
0620 11:42:28.252370 (+ 217us) server_negotiation.cc:299] Negotiation successful
0620 11:42:28.252427 (+ 57us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":51}
I20260620 11:42:28.252655 7328 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 11:42:28.244739 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.234.254:38789 (local address 127.6.234.193:55761)
0620 11:42:28.244905 (+ 166us) negotiation.cc:107] Waiting for socket to connect
0620 11:42:28.244940 (+ 35us) client_negotiation.cc:174] Beginning negotiation
0620 11:42:28.245395 (+ 455us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 11:42:28.245605 (+ 210us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 11:42:28.245608 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 11:42:28.246259 (+ 651us) client_negotiation.cc:189] Negotiated authn=SASL
0620 11:42:28.246447 (+ 188us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 11:42:28.246452 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 11:42:28.247101 (+ 649us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 11:42:28.247101 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 11:42:28.247884 (+ 783us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 11:42:28.247885 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 11:42:28.248214 (+ 329us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 11:42:28.248215 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 11:42:28.248315 (+ 100us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 11:42:28.248829 (+ 514us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 11:42:28.248837 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0620 11:42:28.250194 (+ 1357us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 11:42:28.251484 (+ 1290us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 11:42:28.251489 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 11:42:28.251491 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0620 11:42:28.251700 (+ 209us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 11:42:28.251865 (+ 165us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 11:42:28.251866 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 11:42:28.251866 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0620 11:42:28.251956 (+ 90us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 11:42:28.252158 (+ 202us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 11:42:28.252162 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 11:42:28.252265 (+ 103us) client_negotiation.cc:769] Sending connection context
0620 11:42:28.252346 (+ 81us) client_negotiation.cc:240] Negotiation successful
0620 11:42:28.252454 (+ 108us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":129,"thread_start_us":99,"threads_started":1}
I20260620 11:42:28.253063 7083 external_mini_cluster.cc:1388] Started /tmp/dist-test-task1JRg7N/build/release/bin/kudu as pid 7197
I20260620 11:42:28.253134 7326 heartbeater.cc:347] Connected to a master server at 127.6.234.254:38789
I20260620 11:42:28.253171 7083 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task1JRg7N/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781955746769051-7083-0/minicluster-data/ts-0/wal/instance
I20260620 11:42:28.253219 7326 heartbeater.cc:464] Registering TS with master...
I20260620 11:42:28.253322 7083 external_mini_cluster.cc:1428] Setting key ae67330332a8b8825511bfe8ab3b6191
I20260620 11:42:28.253403 7326 heartbeater.cc:511] Master 127.6.234.254:38789 requested a full tablet report, sending...
I20260620 11:42:28.253965 7134 ts_manager.cc:194] Registered new tserver with Master: 1a68e21b73f6481b87c885c10e245673 (127.6.234.193:44701)
I20260620 11:42:28.254580 7134 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.6.234.193@KRBTEST.COM'} at 127.6.234.193:55761
WARNING: no policy specified for kudu/127.6.234.194@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.6.234.194@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.6.234.194 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task1JRg7N/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781955746769051-7083-0/krb5kdc/kudu_127.6.234.194.keytab.
Entry for principal kudu/127.6.234.194 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task1JRg7N/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781955746769051-7083-0/krb5kdc/kudu_127.6.234.194.keytab.
I20260620 11:42:28.276438 7083 external_mini_cluster.cc:1326] Running /tmp/dist-test-task1JRg7N/build/release/bin/kudu
/tmp/dist-test-task1JRg7N/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task1JRg7N/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781955746769051-7083-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-task1JRg7N/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781955746769051-7083-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task1JRg7N/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781955746769051-7083-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-task1JRg7N/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781955746769051-7083-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.6.234.194:0
--local_ip_for_outbound_sockets=127.6.234.194
--webserver_interface=127.6.234.194
--webserver_port=0
--tserver_master_addrs=127.6.234.254:38789
--keytab_file=/tmp/dist-test-task1JRg7N/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781955746769051-7083-0/krb5kdc/kudu_127.6.234.194.keytab
--principal=kudu/127.6.234.194
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.6.234.212:41303
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_enforce_access_control with env {KRB5CCNAME=/tmp/dist-test-task1JRg7N/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781955746769051-7083-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task1JRg7N/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781955746769051-7083-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task1JRg7N/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781955746769051-7083-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 11:42:28.357656 7331 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 11:42:28.357900 7331 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 11:42:28.357924 7331 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 11:42:28.359133 7331 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 11:42:28.359174 7331 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 11:42:28.359201 7331 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.6.234.194
I20260620 11:42:28.360724 7331 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.6.234.212:41303
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task1JRg7N/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781955746769051-7083-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-task1JRg7N/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781955746769051-7083-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task1JRg7N/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781955746769051-7083-0/krb5kdc/kudu_127.6.234.194.keytab
--principal=kudu/127.6.234.194
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.6.234.194:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task1JRg7N/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781955746769051-7083-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.6.234.194
--webserver_port=0
--tserver_master_addrs=127.6.234.254:38789
--tserver_enforce_access_control=true
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.7331
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.6.234.194
--log_dir=/tmp/dist-test-task1JRg7N/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781955746769051-7083-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 20 Jun 2026 10:32:37 UTC on ccycloud.quasar-bgyofv.root.comops.site
build id 4457173
I20260620 11:42:28.360945 7331 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 11:42:28.361186 7331 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260620 11:42:28.366267 7331 server_base.cc:1034] running on GCE node
W20260620 11:42:28.366245 7339 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 11:42:28.366312 7337 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 11:42:28.366447 7336 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 11:42:28.366676 7331 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 11:42:28.366914 7331 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 11:42:28.368072 7331 hybrid_clock.cc:648] HybridClock initialized: now 1781955748368053 us; error 30 us; skew 500 ppm
Jun 20 11:42:28 dist-test-slave-cjc6 krb5kdc[7090](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781955748, etypes {rep=17 tkt=17 ses=17}, kudu/127.6.234.194@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 11:42:28.370482 7331 init.cc:377] Logged in from keytab as kudu/127.6.234.194@KRBTEST.COM (short username kudu)
I20260620 11:42:28.371061 7331 webserver.cc:460] Webserver started at http://127.6.234.194:40957/ using document root <none> and password file <none>
I20260620 11:42:28.371356 7331 fs_manager.cc:359] Metadata directory not provided
I20260620 11:42:28.371443 7331 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 11:42:28.371596 7331 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 11:42:28.372748 7331 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task1JRg7N/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781955746769051-7083-0/minicluster-data/ts-1/data/instance:
uuid: "d350acc82a2d4c9d9bda4d4662eb9321"
format_stamp: "Formatted at 2026-06-20 11:42:28 on dist-test-slave-cjc6"
server_key: "60456fb82e3bf60a45e48b74b6396f0e"
server_key_iv: "9b816bf1ba0aa0a065580a336c07db2a"
server_key_version: "encryptionkey@0"
I20260620 11:42:28.373126 7331 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task1JRg7N/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781955746769051-7083-0/minicluster-data/ts-1/wal/instance:
uuid: "d350acc82a2d4c9d9bda4d4662eb9321"
format_stamp: "Formatted at 2026-06-20 11:42:28 on dist-test-slave-cjc6"
server_key: "60456fb82e3bf60a45e48b74b6396f0e"
server_key_iv: "9b816bf1ba0aa0a065580a336c07db2a"
server_key_version: "encryptionkey@0"
I20260620 11:42:28.374449 7331 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.003s
I20260620 11:42:28.375133 7346 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 11:42:28.375294 7331 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.001s
I20260620 11:42:28.375381 7331 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task1JRg7N/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781955746769051-7083-0/minicluster-data/ts-1/data,/tmp/dist-test-task1JRg7N/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781955746769051-7083-0/minicluster-data/ts-1/wal
uuid: "d350acc82a2d4c9d9bda4d4662eb9321"
format_stamp: "Formatted at 2026-06-20 11:42:28 on dist-test-slave-cjc6"
server_key: "60456fb82e3bf60a45e48b74b6396f0e"
server_key_iv: "9b816bf1ba0aa0a065580a336c07db2a"
server_key_version: "encryptionkey@0"
I20260620 11:42:28.375461 7331 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task1JRg7N/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781955746769051-7083-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-task1JRg7N/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781955746769051-7083-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-task1JRg7N/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781955746769051-7083-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 11:42:28.387374 7331 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 11:42:28.389206 7331 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 11:42:28.389361 7331 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 11:42:28.389596 7331 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 11:42:28.389911 7331 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 11:42:28.389945 7331 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 11:42:28.389995 7331 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 11:42:28.390023 7331 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 11:42:28.395789 7331 rpc_server.cc:307] RPC server started. Bound to: 127.6.234.194:43989
I20260620 11:42:28.395884 7459 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.6.234.194:43989 every 8 connection(s)
I20260620 11:42:28.396217 7331 server_base.cc:1166] Dumped server information to /tmp/dist-test-task1JRg7N/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781955746769051-7083-0/minicluster-data/ts-1/data/info.pb
I20260620 11:42:28.401103 7083 external_mini_cluster.cc:1388] Started /tmp/dist-test-task1JRg7N/build/release/bin/kudu as pid 7331
I20260620 11:42:28.401227 7083 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task1JRg7N/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781955746769051-7083-0/minicluster-data/ts-1/wal/instance
I20260620 11:42:28.401400 7083 external_mini_cluster.cc:1428] Setting key 4a6f45920411dc206fcea15e9c134524
Jun 20 11:42:28 dist-test-slave-cjc6 krb5kdc[7090](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781955748, etypes {rep=17 tkt=17 ses=17}, kudu/127.6.234.194@KRBTEST.COM for kudu/127.6.234.254@KRBTEST.COM
I20260620 11:42:28.406106 7174 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 11:42:28.396954 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.6.234.194:47093 (local address 127.6.234.254:38789)
0620 11:42:28.397035 (+ 81us) server_negotiation.cc:206] Beginning negotiation
0620 11:42:28.397037 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 11:42:28.397228 (+ 191us) server_negotiation.cc:407] Connection header received
0620 11:42:28.397744 (+ 516us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 11:42:28.397746 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 11:42:28.397756 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 11:42:28.397823 (+ 67us) server_negotiation.cc:226] Negotiated authn=SASL
0620 11:42:28.398976 (+ 1153us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 11:42:28.399453 (+ 477us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 11:42:28.400418 (+ 965us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 11:42:28.400647 (+ 229us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 11:42:28.400736 (+ 89us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 11:42:28.403372 (+ 2636us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 11:42:28.403374 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 11:42:28.403374 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 11:42:28.403382 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0620 11:42:28.404713 (+ 1331us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 11:42:28.405109 (+ 396us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 11:42:28.405110 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 11:42:28.405111 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 11:42:28.405169 (+ 58us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 11:42:28.405424 (+ 255us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 11:42:28.405425 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 11:42:28.405425 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 11:42:28.405545 (+ 120us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 11:42:28.405620 (+ 75us) server_negotiation.cc:1035] Waiting for connection context
0620 11:42:28.405930 (+ 310us) server_negotiation.cc:299] Negotiation successful
0620 11:42:28.406013 (+ 83us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":55}
I20260620 11:42:28.406306 7462 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 11:42:28.396939 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.234.254:38789 (local address 127.6.234.194:47093)
0620 11:42:28.397116 (+ 177us) negotiation.cc:107] Waiting for socket to connect
0620 11:42:28.397156 (+ 40us) client_negotiation.cc:174] Beginning negotiation
0620 11:42:28.397627 (+ 471us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 11:42:28.397877 (+ 250us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 11:42:28.397880 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 11:42:28.398431 (+ 551us) client_negotiation.cc:189] Negotiated authn=SASL
0620 11:42:28.398834 (+ 403us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 11:42:28.398840 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 11:42:28.399538 (+ 698us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 11:42:28.399539 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 11:42:28.400341 (+ 802us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 11:42:28.400343 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 11:42:28.400751 (+ 408us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 11:42:28.400752 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 11:42:28.400873 (+ 121us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 11:42:28.401499 (+ 626us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 11:42:28.401510 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0620 11:42:28.403003 (+ 1493us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 11:42:28.404798 (+ 1795us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 11:42:28.404805 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 11:42:28.404807 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0620 11:42:28.405031 (+ 224us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 11:42:28.405241 (+ 210us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 11:42:28.405242 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 11:42:28.405242 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0620 11:42:28.405350 (+ 108us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 11:42:28.405609 (+ 259us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 11:42:28.405615 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 11:42:28.405797 (+ 182us) client_negotiation.cc:769] Sending connection context
0620 11:42:28.405908 (+ 111us) client_negotiation.cc:240] Negotiation successful
0620 11:42:28.406073 (+ 165us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":126,"thread_start_us":96,"threads_started":1}
I20260620 11:42:28.406761 7460 heartbeater.cc:347] Connected to a master server at 127.6.234.254:38789
I20260620 11:42:28.406870 7460 heartbeater.cc:464] Registering TS with master...
I20260620 11:42:28.407076 7460 heartbeater.cc:511] Master 127.6.234.254:38789 requested a full tablet report, sending...
I20260620 11:42:28.407454 7134 ts_manager.cc:194] Registered new tserver with Master: d350acc82a2d4c9d9bda4d4662eb9321 (127.6.234.194:43989)
I20260620 11:42:28.407825 7134 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.6.234.194@KRBTEST.COM'} at 127.6.234.194:47093
WARNING: no policy specified for kudu/127.6.234.195@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.6.234.195@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.6.234.195 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task1JRg7N/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781955746769051-7083-0/krb5kdc/kudu_127.6.234.195.keytab.
Entry for principal kudu/127.6.234.195 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task1JRg7N/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781955746769051-7083-0/krb5kdc/kudu_127.6.234.195.keytab.
I20260620 11:42:28.424340 7083 external_mini_cluster.cc:1326] Running /tmp/dist-test-task1JRg7N/build/release/bin/kudu
/tmp/dist-test-task1JRg7N/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task1JRg7N/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781955746769051-7083-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-task1JRg7N/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781955746769051-7083-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task1JRg7N/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781955746769051-7083-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-task1JRg7N/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781955746769051-7083-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.6.234.195:0
--local_ip_for_outbound_sockets=127.6.234.195
--webserver_interface=127.6.234.195
--webserver_port=0
--tserver_master_addrs=127.6.234.254:38789
--keytab_file=/tmp/dist-test-task1JRg7N/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781955746769051-7083-0/krb5kdc/kudu_127.6.234.195.keytab
--principal=kudu/127.6.234.195
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.6.234.212:41303
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_enforce_access_control with env {KRB5CCNAME=/tmp/dist-test-task1JRg7N/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781955746769051-7083-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task1JRg7N/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781955746769051-7083-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task1JRg7N/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781955746769051-7083-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 11:42:28.504236 7465 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 11:42:28.504417 7465 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 11:42:28.504439 7465 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 11:42:28.505488 7465 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 11:42:28.505520 7465 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 11:42:28.505546 7465 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.6.234.195
I20260620 11:42:28.506973 7465 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.6.234.212:41303
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task1JRg7N/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781955746769051-7083-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-task1JRg7N/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781955746769051-7083-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task1JRg7N/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781955746769051-7083-0/krb5kdc/kudu_127.6.234.195.keytab
--principal=kudu/127.6.234.195
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.6.234.195:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task1JRg7N/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781955746769051-7083-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.6.234.195
--webserver_port=0
--tserver_master_addrs=127.6.234.254:38789
--tserver_enforce_access_control=true
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.7465
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.6.234.195
--log_dir=/tmp/dist-test-task1JRg7N/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781955746769051-7083-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 20 Jun 2026 10:32:37 UTC on ccycloud.quasar-bgyofv.root.comops.site
build id 4457173
I20260620 11:42:28.507144 7465 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 11:42:28.507304 7465 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 11:42:28.512197 7473 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 11:42:28.512195 7471 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 11:42:28.512217 7470 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 11:42:28.512456 7465 server_base.cc:1034] running on GCE node
I20260620 11:42:28.512660 7465 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 11:42:28.512887 7465 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 11:42:28.514041 7465 hybrid_clock.cc:648] HybridClock initialized: now 1781955748514042 us; error 41 us; skew 500 ppm
Jun 20 11:42:28 dist-test-slave-cjc6 krb5kdc[7090](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781955748, etypes {rep=17 tkt=17 ses=17}, kudu/127.6.234.195@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 11:42:28.516418 7465 init.cc:377] Logged in from keytab as kudu/127.6.234.195@KRBTEST.COM (short username kudu)
I20260620 11:42:28.516978 7465 webserver.cc:460] Webserver started at http://127.6.234.195:38519/ using document root <none> and password file <none>
I20260620 11:42:28.517210 7465 fs_manager.cc:359] Metadata directory not provided
I20260620 11:42:28.517277 7465 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 11:42:28.517405 7465 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 11:42:28.518424 7465 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task1JRg7N/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781955746769051-7083-0/minicluster-data/ts-2/data/instance:
uuid: "d0c7c1e66d51476591f2a104b1f1045c"
format_stamp: "Formatted at 2026-06-20 11:42:28 on dist-test-slave-cjc6"
server_key: "d6cc6b086f41516ac61229292b1cc300"
server_key_iv: "05910880e90d49c3048dde214dec5a2e"
server_key_version: "encryptionkey@0"
I20260620 11:42:28.518850 7465 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task1JRg7N/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781955746769051-7083-0/minicluster-data/ts-2/wal/instance:
uuid: "d0c7c1e66d51476591f2a104b1f1045c"
format_stamp: "Formatted at 2026-06-20 11:42:28 on dist-test-slave-cjc6"
server_key: "d6cc6b086f41516ac61229292b1cc300"
server_key_iv: "05910880e90d49c3048dde214dec5a2e"
server_key_version: "encryptionkey@0"
I20260620 11:42:28.520112 7465 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.002s
I20260620 11:42:28.520797 7480 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 11:42:28.520967 7465 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.001s
I20260620 11:42:28.521060 7465 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task1JRg7N/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781955746769051-7083-0/minicluster-data/ts-2/data,/tmp/dist-test-task1JRg7N/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781955746769051-7083-0/minicluster-data/ts-2/wal
uuid: "d0c7c1e66d51476591f2a104b1f1045c"
format_stamp: "Formatted at 2026-06-20 11:42:28 on dist-test-slave-cjc6"
server_key: "d6cc6b086f41516ac61229292b1cc300"
server_key_iv: "05910880e90d49c3048dde214dec5a2e"
server_key_version: "encryptionkey@0"
I20260620 11:42:28.521144 7465 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task1JRg7N/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781955746769051-7083-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-task1JRg7N/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781955746769051-7083-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-task1JRg7N/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781955746769051-7083-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 11:42:28.533640 7465 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 11:42:28.536019 7465 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 11:42:28.536208 7465 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 11:42:28.536432 7465 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 11:42:28.536674 7465 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 11:42:28.536705 7465 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 11:42:28.536725 7465 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 11:42:28.536738 7465 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 11:42:28.542939 7465 rpc_server.cc:307] RPC server started. Bound to: 127.6.234.195:35541
I20260620 11:42:28.542979 7593 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.6.234.195:35541 every 8 connection(s)
I20260620 11:42:28.543274 7465 server_base.cc:1166] Dumped server information to /tmp/dist-test-task1JRg7N/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781955746769051-7083-0/minicluster-data/ts-2/data/info.pb
I20260620 11:42:28.548575 7083 external_mini_cluster.cc:1388] Started /tmp/dist-test-task1JRg7N/build/release/bin/kudu as pid 7465
I20260620 11:42:28.548694 7083 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task1JRg7N/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781955746769051-7083-0/minicluster-data/ts-2/wal/instance
I20260620 11:42:28.548887 7083 external_mini_cluster.cc:1428] Setting key fce64122456b7b40ec3803030136e92a
Jun 20 11:42:28 dist-test-slave-cjc6 krb5kdc[7090](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781955748, etypes {rep=17 tkt=17 ses=17}, kudu/127.6.234.195@KRBTEST.COM for kudu/127.6.234.254@KRBTEST.COM
I20260620 11:42:28.552451 7174 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 11:42:28.544000 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.6.234.195:46241 (local address 127.6.234.254:38789)
0620 11:42:28.544064 (+ 64us) server_negotiation.cc:206] Beginning negotiation
0620 11:42:28.544066 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 11:42:28.544233 (+ 167us) server_negotiation.cc:407] Connection header received
0620 11:42:28.544701 (+ 468us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 11:42:28.544702 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 11:42:28.544714 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 11:42:28.544781 (+ 67us) server_negotiation.cc:226] Negotiated authn=SASL
0620 11:42:28.545641 (+ 860us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 11:42:28.546207 (+ 566us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 11:42:28.547138 (+ 931us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 11:42:28.547327 (+ 189us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 11:42:28.547385 (+ 58us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 11:42:28.549659 (+ 2274us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 11:42:28.549661 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 11:42:28.549665 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 11:42:28.549672 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0620 11:42:28.551099 (+ 1427us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 11:42:28.551517 (+ 418us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 11:42:28.551518 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 11:42:28.551519 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 11:42:28.551575 (+ 56us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 11:42:28.551822 (+ 247us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 11:42:28.551823 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 11:42:28.551824 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 11:42:28.551941 (+ 117us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 11:42:28.552013 (+ 72us) server_negotiation.cc:1035] Waiting for connection context
0620 11:42:28.552284 (+ 271us) server_negotiation.cc:299] Negotiation successful
0620 11:42:28.552357 (+ 73us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":40}
I20260620 11:42:28.552645 7596 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 11:42:28.543948 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.234.254:38789 (local address 127.6.234.195:46241)
0620 11:42:28.544096 (+ 148us) negotiation.cc:107] Waiting for socket to connect
0620 11:42:28.544136 (+ 40us) client_negotiation.cc:174] Beginning negotiation
0620 11:42:28.544595 (+ 459us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 11:42:28.544831 (+ 236us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 11:42:28.544834 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 11:42:28.545367 (+ 533us) client_negotiation.cc:189] Negotiated authn=SASL
0620 11:42:28.545550 (+ 183us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 11:42:28.545554 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 11:42:28.546285 (+ 731us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 11:42:28.546286 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 11:42:28.547062 (+ 776us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 11:42:28.547064 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 11:42:28.547400 (+ 336us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 11:42:28.547402 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 11:42:28.547494 (+ 92us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 11:42:28.548005 (+ 511us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 11:42:28.548019 (+ 14us) client_negotiation.cc:656] calling sasl_client_start()
0620 11:42:28.549558 (+ 1539us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 11:42:28.551198 (+ 1640us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 11:42:28.551205 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 11:42:28.551207 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0620 11:42:28.551428 (+ 221us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 11:42:28.551642 (+ 214us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 11:42:28.551643 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 11:42:28.551644 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0620 11:42:28.551754 (+ 110us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 11:42:28.552019 (+ 265us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 11:42:28.552026 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 11:42:28.552156 (+ 130us) client_negotiation.cc:769] Sending connection context
0620 11:42:28.552263 (+ 107us) client_negotiation.cc:240] Negotiation successful
0620 11:42:28.552426 (+ 163us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":111,"thread_start_us":85,"threads_started":1}
I20260620 11:42:28.553072 7594 heartbeater.cc:347] Connected to a master server at 127.6.234.254:38789
I20260620 11:42:28.553148 7594 heartbeater.cc:464] Registering TS with master...
I20260620 11:42:28.553323 7594 heartbeater.cc:511] Master 127.6.234.254:38789 requested a full tablet report, sending...
I20260620 11:42:28.553668 7134 ts_manager.cc:194] Registered new tserver with Master: d0c7c1e66d51476591f2a104b1f1045c (127.6.234.195:35541)
I20260620 11:42:28.554072 7134 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.6.234.195@KRBTEST.COM'} at 127.6.234.195:46241
I20260620 11:42:28.560972 7083 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
Jun 20 11:42:28 dist-test-slave-cjc6 krb5kdc[7090](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781955748, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-user@KRBTEST.COM:
Jun 20 11:42:28 dist-test-slave-cjc6 krb5kdc[7090](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781955748, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for kudu/127.6.234.193@KRBTEST.COM
I20260620 11:42:28.583683 7601 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 11:42:28.575937 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59450 (local address 127.6.234.193:44701)
0620 11:42:28.576109 (+ 172us) server_negotiation.cc:206] Beginning negotiation
0620 11:42:28.576113 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 11:42:28.576123 (+ 10us) server_negotiation.cc:407] Connection header received
0620 11:42:28.576148 (+ 25us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 11:42:28.576150 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 11:42:28.576163 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 11:42:28.576227 (+ 64us) server_negotiation.cc:226] Negotiated authn=SASL
0620 11:42:28.576743 (+ 516us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 11:42:28.577488 (+ 745us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 11:42:28.578358 (+ 870us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 11:42:28.578717 (+ 359us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 11:42:28.578775 (+ 58us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 11:42:28.580995 (+ 2220us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 11:42:28.581002 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 11:42:28.581003 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 11:42:28.581009 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0620 11:42:28.582635 (+ 1626us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 11:42:28.582971 (+ 336us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 11:42:28.582973 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 11:42:28.582974 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 11:42:28.583028 (+ 54us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 11:42:28.583192 (+ 164us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 11:42:28.583192 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 11:42:28.583192 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 11:42:28.583312 (+ 120us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 11:42:28.583413 (+ 101us) server_negotiation.cc:1035] Waiting for connection context
0620 11:42:28.583491 (+ 78us) server_negotiation.cc:299] Negotiation successful
0620 11:42:28.583551 (+ 60us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":135,"thread_start_us":84,"threads_started":1}
W20260620 11:42:28.584081 7240 server_base.cc:1116] Unauthorized access attempt to method kudu.tserver.TabletServerService.ListTablets from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:59450
Jun 20 11:42:28 dist-test-slave-cjc6 krb5kdc[7090](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781955748, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
Jun 20 11:42:28 dist-test-slave-cjc6 krb5kdc[7090](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781955748, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.6.234.193@KRBTEST.COM
I20260620 11:42:28.606141 7601 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 11:42:28.599521 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59462 (local address 127.6.234.193:44701)
0620 11:42:28.599681 (+ 160us) server_negotiation.cc:206] Beginning negotiation
0620 11:42:28.599683 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 11:42:28.599725 (+ 42us) server_negotiation.cc:407] Connection header received
0620 11:42:28.599807 (+ 82us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 11:42:28.599808 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 11:42:28.599823 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 11:42:28.599891 (+ 68us) server_negotiation.cc:226] Negotiated authn=SASL
0620 11:42:28.600423 (+ 532us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 11:42:28.600906 (+ 483us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 11:42:28.601648 (+ 742us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 11:42:28.601910 (+ 262us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 11:42:28.601995 (+ 85us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 11:42:28.604043 (+ 2048us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 11:42:28.604044 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 11:42:28.604044 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 11:42:28.604050 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0620 11:42:28.605197 (+ 1147us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 11:42:28.605497 (+ 300us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 11:42:28.605497 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 11:42:28.605498 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 11:42:28.605543 (+ 45us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 11:42:28.605742 (+ 199us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 11:42:28.605743 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 11:42:28.605743 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 11:42:28.605841 (+ 98us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 11:42:28.605929 (+ 88us) server_negotiation.cc:1035] Waiting for connection context
0620 11:42:28.606016 (+ 87us) server_negotiation.cc:299] Negotiation successful
0620 11:42:28.606072 (+ 56us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":137}
I20260620 11:42:28.606669 7083 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task1JRg7N/build/release/bin/kudu with pid 7197
I20260620 11:42:28.611315 7083 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task1JRg7N/build/release/bin/kudu with pid 7331
I20260620 11:42:28.615439 7083 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task1JRg7N/build/release/bin/kudu with pid 7465
I20260620 11:42:28.619378 7083 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task1JRg7N/build/release/bin/kudu with pid 7103
2026-06-20T11:42:28Z chronyd exiting
[ OK ] SecurityITest.TestAuthorizationOnListTablets (1858 ms)
[ RUN ] SecurityITest.TestAuthorizationOnChecksum
Loading random data
Initializing database '/tmp/dist-test-task1JRg7N/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781955746769051-7083-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 20 11:42:28 dist-test-slave-cjc6 krb5kdc[7608](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 20 11:42:28 dist-test-slave-cjc6 krb5kdc[7608](info): set up 2 sockets
Jun 20 11:42:28 dist-test-slave-cjc6 krb5kdc[7608](info): commencing operation
krb5kdc: starting...
W20260620 11:42:29.661141 7083 mini_kdc.cc:121] Time spent starting KDC: real 1.022s user 0.000s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task1JRg7N/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781955746769051-7083-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task1JRg7N/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781955746769051-7083-0/krb5kdc/test-user.keytab.
Jun 20 11:42:29 dist-test-slave-cjc6 krb5kdc[7608](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781955749, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-20T11:42:29Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-20T11:42:29Z Disabled control of system clock
WARNING: no policy specified for kudu/127.6.234.254@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.6.234.254@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.6.234.254 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task1JRg7N/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781955746769051-7083-0/krb5kdc/kudu_127.6.234.254.keytab.
Entry for principal kudu/127.6.234.254 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task1JRg7N/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781955746769051-7083-0/krb5kdc/kudu_127.6.234.254.keytab.
I20260620 11:42:29.777526 7083 external_mini_cluster.cc:1326] Running /tmp/dist-test-task1JRg7N/build/release/bin/kudu
/tmp/dist-test-task1JRg7N/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task1JRg7N/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781955746769051-7083-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task1JRg7N/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781955746769051-7083-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task1JRg7N/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781955746769051-7083-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task1JRg7N/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781955746769051-7083-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.6.234.254:45361
--webserver_interface=127.6.234.254
--webserver_port=0
--keytab_file=/tmp/dist-test-task1JRg7N/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781955746769051-7083-0/krb5kdc/kudu_127.6.234.254.keytab
--principal=kudu/127.6.234.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.6.234.212:40445
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.6.234.254:45361
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task1JRg7N/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781955746769051-7083-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task1JRg7N/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781955746769051-7083-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task1JRg7N/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781955746769051-7083-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 11:42:29.857439 7621 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 11:42:29.857692 7621 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 11:42:29.857751 7621 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 11:42:29.858914 7621 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 11:42:29.858989 7621 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 11:42:29.859014 7621 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 11:42:29.859035 7621 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 11:42:29.859063 7621 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 11:42:29.860525 7621 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.6.234.212:40445
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task1JRg7N/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781955746769051-7083-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task1JRg7N/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781955746769051-7083-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.6.234.254:45361
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task1JRg7N/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781955746769051-7083-0/krb5kdc/kudu_127.6.234.254.keytab
--principal=kudu/127.6.234.254
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.6.234.254:45361
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task1JRg7N/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781955746769051-7083-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.6.234.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.7621
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task1JRg7N/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781955746769051-7083-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 20 Jun 2026 10:32:37 UTC on ccycloud.quasar-bgyofv.root.comops.site
build id 4457173
I20260620 11:42:29.860803 7621 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 11:42:29.861042 7621 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 11:42:29.865654 7629 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 11:42:29.865760 7621 server_base.cc:1034] running on GCE node
W20260620 11:42:29.867004 7626 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 11:42:29.867134 7627 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 11:42:29.867417 7621 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 11:42:29.867702 7621 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 11:42:29.868870 7621 hybrid_clock.cc:648] HybridClock initialized: now 1781955749868834 us; error 66 us; skew 500 ppm
Jun 20 11:42:29 dist-test-slave-cjc6 krb5kdc[7608](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781955749, etypes {rep=17 tkt=17 ses=17}, kudu/127.6.234.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 11:42:29.871369 7621 init.cc:377] Logged in from keytab as kudu/127.6.234.254@KRBTEST.COM (short username kudu)
I20260620 11:42:29.871845 7621 webserver.cc:460] Webserver started at http://127.6.234.254:43997/ using document root <none> and password file <none>
I20260620 11:42:29.872094 7621 fs_manager.cc:359] Metadata directory not provided
I20260620 11:42:29.872164 7621 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 11:42:29.872316 7621 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 11:42:29.873374 7621 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task1JRg7N/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781955746769051-7083-0/minicluster-data/master-0/data/instance:
uuid: "b968e23876e9400d8fdba96495b52ca4"
format_stamp: "Formatted at 2026-06-20 11:42:29 on dist-test-slave-cjc6"
server_key: "7499e5e31f8e539f4521d4d72a556172"
server_key_iv: "3e9c0372509a8c06ee80effb9884b884"
server_key_version: "encryptionkey@0"
I20260620 11:42:29.873803 7621 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task1JRg7N/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781955746769051-7083-0/minicluster-data/master-0/wal/instance:
uuid: "b968e23876e9400d8fdba96495b52ca4"
format_stamp: "Formatted at 2026-06-20 11:42:29 on dist-test-slave-cjc6"
server_key: "7499e5e31f8e539f4521d4d72a556172"
server_key_iv: "3e9c0372509a8c06ee80effb9884b884"
server_key_version: "encryptionkey@0"
I20260620 11:42:29.875090 7621 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.001s
I20260620 11:42:29.875787 7636 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 11:42:29.875949 7621 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.001s
I20260620 11:42:29.876048 7621 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task1JRg7N/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781955746769051-7083-0/minicluster-data/master-0/data,/tmp/dist-test-task1JRg7N/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781955746769051-7083-0/minicluster-data/master-0/wal
uuid: "b968e23876e9400d8fdba96495b52ca4"
format_stamp: "Formatted at 2026-06-20 11:42:29 on dist-test-slave-cjc6"
server_key: "7499e5e31f8e539f4521d4d72a556172"
server_key_iv: "3e9c0372509a8c06ee80effb9884b884"
server_key_version: "encryptionkey@0"
I20260620 11:42:29.876154 7621 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task1JRg7N/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781955746769051-7083-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task1JRg7N/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781955746769051-7083-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task1JRg7N/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781955746769051-7083-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 11:42:29.908246 7621 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 11:42:29.910276 7621 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 11:42:29.910451 7621 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 11:42:29.914335 7621 rpc_server.cc:307] RPC server started. Bound to: 127.6.234.254:45361
I20260620 11:42:29.914374 7688 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.6.234.254:45361 every 8 connection(s)
I20260620 11:42:29.914746 7621 server_base.cc:1166] Dumped server information to /tmp/dist-test-task1JRg7N/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781955746769051-7083-0/minicluster-data/master-0/data/info.pb
I20260620 11:42:29.915565 7689 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 11:42:29.917688 7689 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P b968e23876e9400d8fdba96495b52ca4: Bootstrap starting.
I20260620 11:42:29.918210 7689 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P b968e23876e9400d8fdba96495b52ca4: Neither blocks nor log segments found. Creating new log.
I20260620 11:42:29.918438 7689 log.cc:824] T 00000000000000000000000000000000 P b968e23876e9400d8fdba96495b52ca4: Log is configured to *not* fsync() on all Append() calls
I20260620 11:42:29.919066 7689 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P b968e23876e9400d8fdba96495b52ca4: No bootstrap required, opened a new log
I20260620 11:42:29.920549 7689 raft_consensus.cc:348] T 00000000000000000000000000000000 P b968e23876e9400d8fdba96495b52ca4 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b968e23876e9400d8fdba96495b52ca4" member_type: VOTER last_known_addr { host: "127.6.234.254" port: 45361 } }
I20260620 11:42:29.920686 7689 raft_consensus.cc:374] T 00000000000000000000000000000000 P b968e23876e9400d8fdba96495b52ca4 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 11:42:29.920707 7689 raft_consensus.cc:729] T 00000000000000000000000000000000 P b968e23876e9400d8fdba96495b52ca4 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b968e23876e9400d8fdba96495b52ca4, State: Initialized, Role: FOLLOWER
I20260620 11:42:29.920851 7689 consensus_queue.cc:260] T 00000000000000000000000000000000 P b968e23876e9400d8fdba96495b52ca4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b968e23876e9400d8fdba96495b52ca4" member_type: VOTER last_known_addr { host: "127.6.234.254" port: 45361 } }
I20260620 11:42:29.920929 7689 raft_consensus.cc:388] T 00000000000000000000000000000000 P b968e23876e9400d8fdba96495b52ca4 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260620 11:42:29.920961 7689 raft_consensus.cc:482] T 00000000000000000000000000000000 P b968e23876e9400d8fdba96495b52ca4 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260620 11:42:29.920990 7689 raft_consensus.cc:3037] T 00000000000000000000000000000000 P b968e23876e9400d8fdba96495b52ca4 [term 0 FOLLOWER]: Advancing to term 1
I20260620 11:42:29.921428 7689 raft_consensus.cc:504] T 00000000000000000000000000000000 P b968e23876e9400d8fdba96495b52ca4 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b968e23876e9400d8fdba96495b52ca4" member_type: VOTER last_known_addr { host: "127.6.234.254" port: 45361 } }
I20260620 11:42:29.921518 7689 leader_election.cc:302] T 00000000000000000000000000000000 P b968e23876e9400d8fdba96495b52ca4 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: b968e23876e9400d8fdba96495b52ca4; no voters:
I20260620 11:42:29.921660 7689 leader_election.cc:288] T 00000000000000000000000000000000 P b968e23876e9400d8fdba96495b52ca4 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260620 11:42:29.921723 7692 raft_consensus.cc:2781] T 00000000000000000000000000000000 P b968e23876e9400d8fdba96495b52ca4 [term 1 FOLLOWER]: Leader election won for term 1
I20260620 11:42:29.921851 7692 raft_consensus.cc:686] T 00000000000000000000000000000000 P b968e23876e9400d8fdba96495b52ca4 [term 1 LEADER]: Becoming Leader. State: Replica: b968e23876e9400d8fdba96495b52ca4, State: Running, Role: LEADER
I20260620 11:42:29.922022 7083 external_mini_cluster.cc:1388] Started /tmp/dist-test-task1JRg7N/build/release/bin/kudu as pid 7621
I20260620 11:42:29.922015 7692 consensus_queue.cc:237] T 00000000000000000000000000000000 P b968e23876e9400d8fdba96495b52ca4 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b968e23876e9400d8fdba96495b52ca4" member_type: VOTER last_known_addr { host: "127.6.234.254" port: 45361 } }
I20260620 11:42:29.922101 7083 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task1JRg7N/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781955746769051-7083-0/minicluster-data/master-0/wal/instance
I20260620 11:42:29.922269 7083 external_mini_cluster.cc:1428] Setting key 5eb3cfc935a479b56f0bfefd007f4b58
I20260620 11:42:29.922549 7689 sys_catalog.cc:564] T 00000000000000000000000000000000 P b968e23876e9400d8fdba96495b52ca4 [sys.catalog]: configured and running, proceeding with master startup.
I20260620 11:42:29.922528 7694 sys_catalog.cc:455] T 00000000000000000000000000000000 P b968e23876e9400d8fdba96495b52ca4 [sys.catalog]: SysCatalogTable state changed. Reason: New leader b968e23876e9400d8fdba96495b52ca4. Latest consensus state: current_term: 1 leader_uuid: "b968e23876e9400d8fdba96495b52ca4" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b968e23876e9400d8fdba96495b52ca4" member_type: VOTER last_known_addr { host: "127.6.234.254" port: 45361 } } }
I20260620 11:42:29.922670 7694 sys_catalog.cc:458] T 00000000000000000000000000000000 P b968e23876e9400d8fdba96495b52ca4 [sys.catalog]: This master's current role is: LEADER
I20260620 11:42:29.922626 7693 sys_catalog.cc:455] T 00000000000000000000000000000000 P b968e23876e9400d8fdba96495b52ca4 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "b968e23876e9400d8fdba96495b52ca4" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b968e23876e9400d8fdba96495b52ca4" member_type: VOTER last_known_addr { host: "127.6.234.254" port: 45361 } } }
I20260620 11:42:29.922969 7693 sys_catalog.cc:458] T 00000000000000000000000000000000 P b968e23876e9400d8fdba96495b52ca4 [sys.catalog]: This master's current role is: LEADER
I20260620 11:42:29.923113 7696 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260620 11:42:29.923664 7696 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260620 11:42:29.925354 7696 catalog_manager.cc:1347] Generated new cluster ID: 176c5cce98a1460fb67aa268f97f1684
I20260620 11:42:29.925403 7696 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
Jun 20 11:42:29 dist-test-slave-cjc6 krb5kdc[7608](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781955749, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.6.234.254@KRBTEST.COM
I20260620 11:42:29.932260 7696 catalog_manager.cc:1370] Generated new certificate authority record
I20260620 11:42:29.932727 7696 catalog_manager.cc:1504] Loading token signing keys...
I20260620 11:42:29.934464 7699 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 11:42:29.922950 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53036 (local address 127.6.234.254:45361)
0620 11:42:29.923127 (+ 177us) server_negotiation.cc:206] Beginning negotiation
0620 11:42:29.923138 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0620 11:42:29.923170 (+ 32us) server_negotiation.cc:407] Connection header received
0620 11:42:29.923292 (+ 122us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 11:42:29.923301 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 11:42:29.923365 (+ 64us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 11:42:29.923541 (+ 176us) server_negotiation.cc:226] Negotiated authn=SASL
0620 11:42:29.924297 (+ 756us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 11:42:29.925534 (+ 1237us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 11:42:29.926612 (+ 1078us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 11:42:29.927204 (+ 592us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 11:42:29.927303 (+ 99us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 11:42:29.930088 (+ 2785us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 11:42:29.930093 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 11:42:29.930094 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 11:42:29.930108 (+ 14us) server_negotiation.cc:922] calling sasl_server_start()
0620 11:42:29.932781 (+ 2673us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 11:42:29.933203 (+ 422us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 11:42:29.933206 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 11:42:29.933206 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 11:42:29.933321 (+ 115us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 11:42:29.933542 (+ 221us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 11:42:29.933543 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 11:42:29.933543 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 11:42:29.933836 (+ 293us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 11:42:29.933987 (+ 151us) server_negotiation.cc:1035] Waiting for connection context
0620 11:42:29.934117 (+ 130us) server_negotiation.cc:299] Negotiation successful
0620 11:42:29.934255 (+ 138us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":132,"thread_start_us":86,"threads_started":1}
I20260620 11:42:29.949468 7696 catalog_manager.cc:5868] T 00000000000000000000000000000000 P b968e23876e9400d8fdba96495b52ca4: Generated new TSK 0
I20260620 11:42:29.949757 7696 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.6.234.193@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.6.234.193@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.6.234.193 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task1JRg7N/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781955746769051-7083-0/krb5kdc/kudu_127.6.234.193.keytab.
Entry for principal kudu/127.6.234.193 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task1JRg7N/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781955746769051-7083-0/krb5kdc/kudu_127.6.234.193.keytab.
I20260620 11:42:29.976351 7083 external_mini_cluster.cc:1326] Running /tmp/dist-test-task1JRg7N/build/release/bin/kudu
/tmp/dist-test-task1JRg7N/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task1JRg7N/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781955746769051-7083-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-task1JRg7N/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781955746769051-7083-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task1JRg7N/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781955746769051-7083-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-task1JRg7N/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781955746769051-7083-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.6.234.193:0
--local_ip_for_outbound_sockets=127.6.234.193
--webserver_interface=127.6.234.193
--webserver_port=0
--tserver_master_addrs=127.6.234.254:45361
--keytab_file=/tmp/dist-test-task1JRg7N/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781955746769051-7083-0/krb5kdc/kudu_127.6.234.193.keytab
--principal=kudu/127.6.234.193
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.6.234.212:40445
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_enforce_access_control with env {KRB5CCNAME=/tmp/dist-test-task1JRg7N/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781955746769051-7083-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task1JRg7N/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781955746769051-7083-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task1JRg7N/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781955746769051-7083-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 11:42:30.058914 7715 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 11:42:30.059135 7715 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 11:42:30.059159 7715 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 11:42:30.060331 7715 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 11:42:30.060372 7715 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 11:42:30.060400 7715 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.6.234.193
I20260620 11:42:30.061975 7715 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.6.234.212:40445
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task1JRg7N/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781955746769051-7083-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-task1JRg7N/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781955746769051-7083-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task1JRg7N/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781955746769051-7083-0/krb5kdc/kudu_127.6.234.193.keytab
--principal=kudu/127.6.234.193
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.6.234.193:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task1JRg7N/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781955746769051-7083-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.6.234.193
--webserver_port=0
--tserver_master_addrs=127.6.234.254:45361
--tserver_enforce_access_control=true
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.7715
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.6.234.193
--log_dir=/tmp/dist-test-task1JRg7N/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781955746769051-7083-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 20 Jun 2026 10:32:37 UTC on ccycloud.quasar-bgyofv.root.comops.site
build id 4457173
I20260620 11:42:30.062263 7715 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 11:42:30.062528 7715 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 11:42:30.067407 7720 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 11:42:30.067449 7721 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 11:42:30.067683 7715 server_base.cc:1034] running on GCE node
W20260620 11:42:30.067566 7723 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 11:42:30.067936 7715 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 11:42:30.068189 7715 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 11:42:30.069351 7715 hybrid_clock.cc:648] HybridClock initialized: now 1781955750069339 us; error 42 us; skew 500 ppm
Jun 20 11:42:30 dist-test-slave-cjc6 krb5kdc[7608](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781955750, etypes {rep=17 tkt=17 ses=17}, kudu/127.6.234.193@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 11:42:30.071907 7715 init.cc:377] Logged in from keytab as kudu/127.6.234.193@KRBTEST.COM (short username kudu)
I20260620 11:42:30.072431 7715 webserver.cc:460] Webserver started at http://127.6.234.193:42413/ using document root <none> and password file <none>
I20260620 11:42:30.072664 7715 fs_manager.cc:359] Metadata directory not provided
I20260620 11:42:30.072732 7715 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 11:42:30.072882 7715 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 11:42:30.073963 7715 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task1JRg7N/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781955746769051-7083-0/minicluster-data/ts-0/data/instance:
uuid: "6409f1c1cd7948bf8191639fe8679991"
format_stamp: "Formatted at 2026-06-20 11:42:30 on dist-test-slave-cjc6"
server_key: "fbad4ece0b6de77b82af405bfdb9fc86"
server_key_iv: "99a07983c9d3d3643d21d4d9d95b7a7f"
server_key_version: "encryptionkey@0"
I20260620 11:42:30.074321 7715 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task1JRg7N/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781955746769051-7083-0/minicluster-data/ts-0/wal/instance:
uuid: "6409f1c1cd7948bf8191639fe8679991"
format_stamp: "Formatted at 2026-06-20 11:42:30 on dist-test-slave-cjc6"
server_key: "fbad4ece0b6de77b82af405bfdb9fc86"
server_key_iv: "99a07983c9d3d3643d21d4d9d95b7a7f"
server_key_version: "encryptionkey@0"
I20260620 11:42:30.075603 7715 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260620 11:42:30.076421 7730 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 11:42:30.076618 7715 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260620 11:42:30.076718 7715 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task1JRg7N/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781955746769051-7083-0/minicluster-data/ts-0/data,/tmp/dist-test-task1JRg7N/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781955746769051-7083-0/minicluster-data/ts-0/wal
uuid: "6409f1c1cd7948bf8191639fe8679991"
format_stamp: "Formatted at 2026-06-20 11:42:30 on dist-test-slave-cjc6"
server_key: "fbad4ece0b6de77b82af405bfdb9fc86"
server_key_iv: "99a07983c9d3d3643d21d4d9d95b7a7f"
server_key_version: "encryptionkey@0"
I20260620 11:42:30.076795 7715 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task1JRg7N/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781955746769051-7083-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-task1JRg7N/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781955746769051-7083-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-task1JRg7N/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781955746769051-7083-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 11:42:30.114164 7715 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 11:42:30.116154 7715 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 11:42:30.116326 7715 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 11:42:30.116580 7715 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 11:42:30.116911 7715 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 11:42:30.116966 7715 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 11:42:30.117033 7715 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 11:42:30.117056 7715 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 11:42:30.122929 7715 rpc_server.cc:307] RPC server started. Bound to: 127.6.234.193:38839
I20260620 11:42:30.122982 7843 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.6.234.193:38839 every 8 connection(s)
I20260620 11:42:30.123307 7715 server_base.cc:1166] Dumped server information to /tmp/dist-test-task1JRg7N/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781955746769051-7083-0/minicluster-data/ts-0/data/info.pb
Jun 20 11:42:30 dist-test-slave-cjc6 krb5kdc[7608](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781955750, etypes {rep=17 tkt=17 ses=17}, kudu/127.6.234.193@KRBTEST.COM for kudu/127.6.234.254@KRBTEST.COM
I20260620 11:42:30.131712 7083 external_mini_cluster.cc:1388] Started /tmp/dist-test-task1JRg7N/build/release/bin/kudu as pid 7715
I20260620 11:42:30.131814 7083 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task1JRg7N/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781955746769051-7083-0/minicluster-data/ts-0/wal/instance
I20260620 11:42:30.132022 7083 external_mini_cluster.cc:1428] Setting key d18764e42147cd51a8856a71d793d6ac
I20260620 11:42:30.132723 7699 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 11:42:30.124077 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.6.234.193:50349 (local address 127.6.234.254:45361)
0620 11:42:30.124145 (+ 68us) server_negotiation.cc:206] Beginning negotiation
0620 11:42:30.124147 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 11:42:30.124355 (+ 208us) server_negotiation.cc:407] Connection header received
0620 11:42:30.125293 (+ 938us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 11:42:30.125294 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 11:42:30.125303 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 11:42:30.125357 (+ 54us) server_negotiation.cc:226] Negotiated authn=SASL
0620 11:42:30.125959 (+ 602us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 11:42:30.126467 (+ 508us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 11:42:30.127427 (+ 960us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 11:42:30.127613 (+ 186us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 11:42:30.127665 (+ 52us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 11:42:30.129903 (+ 2238us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 11:42:30.129905 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 11:42:30.129906 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 11:42:30.129912 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0620 11:42:30.131178 (+ 1266us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 11:42:30.131589 (+ 411us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 11:42:30.131590 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 11:42:30.131590 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 11:42:30.131640 (+ 50us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 11:42:30.131934 (+ 294us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 11:42:30.131935 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 11:42:30.131936 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 11:42:30.132047 (+ 111us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 11:42:30.132139 (+ 92us) server_negotiation.cc:1035] Waiting for connection context
0620 11:42:30.132514 (+ 375us) server_negotiation.cc:299] Negotiation successful
0620 11:42:30.132616 (+ 102us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":40}
I20260620 11:42:30.132913 7846 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 11:42:30.124053 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.234.254:45361 (local address 127.6.234.193:50349)
0620 11:42:30.124229 (+ 176us) negotiation.cc:107] Waiting for socket to connect
0620 11:42:30.124270 (+ 41us) client_negotiation.cc:174] Beginning negotiation
0620 11:42:30.125196 (+ 926us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 11:42:30.125420 (+ 224us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 11:42:30.125423 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 11:42:30.125675 (+ 252us) client_negotiation.cc:189] Negotiated authn=SASL
0620 11:42:30.125888 (+ 213us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 11:42:30.125892 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 11:42:30.126547 (+ 655us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 11:42:30.126547 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 11:42:30.127354 (+ 807us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 11:42:30.127355 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 11:42:30.127687 (+ 332us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 11:42:30.127687 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 11:42:30.127792 (+ 105us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 11:42:30.128351 (+ 559us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 11:42:30.128359 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0620 11:42:30.129818 (+ 1459us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 11:42:30.131255 (+ 1437us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 11:42:30.131261 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 11:42:30.131262 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0620 11:42:30.131514 (+ 252us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 11:42:30.131705 (+ 191us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 11:42:30.131705 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 11:42:30.131706 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0620 11:42:30.131866 (+ 160us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 11:42:30.132166 (+ 300us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 11:42:30.132171 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 11:42:30.132353 (+ 182us) client_negotiation.cc:769] Sending connection context
0620 11:42:30.132491 (+ 138us) client_negotiation.cc:240] Negotiation successful
0620 11:42:30.132656 (+ 165us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":134,"thread_start_us":104,"threads_started":1}
I20260620 11:42:30.133389 7844 heartbeater.cc:347] Connected to a master server at 127.6.234.254:45361
I20260620 11:42:30.133500 7844 heartbeater.cc:464] Registering TS with master...
I20260620 11:42:30.133728 7844 heartbeater.cc:511] Master 127.6.234.254:45361 requested a full tablet report, sending...
I20260620 11:42:30.134264 7653 ts_manager.cc:194] Registered new tserver with Master: 6409f1c1cd7948bf8191639fe8679991 (127.6.234.193:38839)
I20260620 11:42:30.135207 7653 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.6.234.193@KRBTEST.COM'} at 127.6.234.193:50349
WARNING: no policy specified for kudu/127.6.234.194@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.6.234.194@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.6.234.194 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task1JRg7N/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781955746769051-7083-0/krb5kdc/kudu_127.6.234.194.keytab.
Entry for principal kudu/127.6.234.194 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task1JRg7N/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781955746769051-7083-0/krb5kdc/kudu_127.6.234.194.keytab.
I20260620 11:42:30.156791 7083 external_mini_cluster.cc:1326] Running /tmp/dist-test-task1JRg7N/build/release/bin/kudu
/tmp/dist-test-task1JRg7N/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task1JRg7N/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781955746769051-7083-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-task1JRg7N/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781955746769051-7083-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task1JRg7N/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781955746769051-7083-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-task1JRg7N/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781955746769051-7083-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.6.234.194:0
--local_ip_for_outbound_sockets=127.6.234.194
--webserver_interface=127.6.234.194
--webserver_port=0
--tserver_master_addrs=127.6.234.254:45361
--keytab_file=/tmp/dist-test-task1JRg7N/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781955746769051-7083-0/krb5kdc/kudu_127.6.234.194.keytab
--principal=kudu/127.6.234.194
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.6.234.212:40445
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_enforce_access_control with env {KRB5CCNAME=/tmp/dist-test-task1JRg7N/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781955746769051-7083-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task1JRg7N/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781955746769051-7083-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task1JRg7N/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781955746769051-7083-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 11:42:30.236447 7849 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 11:42:30.236650 7849 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 11:42:30.236675 7849 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 11:42:30.237877 7849 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 11:42:30.237918 7849 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 11:42:30.237946 7849 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.6.234.194
I20260620 11:42:30.239439 7849 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.6.234.212:40445
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task1JRg7N/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781955746769051-7083-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-task1JRg7N/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781955746769051-7083-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task1JRg7N/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781955746769051-7083-0/krb5kdc/kudu_127.6.234.194.keytab
--principal=kudu/127.6.234.194
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.6.234.194:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task1JRg7N/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781955746769051-7083-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.6.234.194
--webserver_port=0
--tserver_master_addrs=127.6.234.254:45361
--tserver_enforce_access_control=true
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.7849
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.6.234.194
--log_dir=/tmp/dist-test-task1JRg7N/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781955746769051-7083-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 20 Jun 2026 10:32:37 UTC on ccycloud.quasar-bgyofv.root.comops.site
build id 4457173
I20260620 11:42:30.239674 7849 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 11:42:30.239904 7849 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 11:42:30.244932 7855 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 11:42:30.244939 7854 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 11:42:30.244975 7849 server_base.cc:1034] running on GCE node
W20260620 11:42:30.244951 7857 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 11:42:30.245426 7849 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 11:42:30.245641 7849 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 11:42:30.246781 7849 hybrid_clock.cc:648] HybridClock initialized: now 1781955750246764 us; error 35 us; skew 500 ppm
Jun 20 11:42:30 dist-test-slave-cjc6 krb5kdc[7608](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781955750, etypes {rep=17 tkt=17 ses=17}, kudu/127.6.234.194@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 11:42:30.249373 7849 init.cc:377] Logged in from keytab as kudu/127.6.234.194@KRBTEST.COM (short username kudu)
I20260620 11:42:30.249948 7849 webserver.cc:460] Webserver started at http://127.6.234.194:43753/ using document root <none> and password file <none>
I20260620 11:42:30.250180 7849 fs_manager.cc:359] Metadata directory not provided
I20260620 11:42:30.250227 7849 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 11:42:30.250356 7849 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 11:42:30.251462 7849 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task1JRg7N/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781955746769051-7083-0/minicluster-data/ts-1/data/instance:
uuid: "926a5e7ef3d648f5b24454af0fe49484"
format_stamp: "Formatted at 2026-06-20 11:42:30 on dist-test-slave-cjc6"
server_key: "0260e705983955de7338fc4c214b71b7"
server_key_iv: "3dab60fd74de29c5d2347493c33d3ec3"
server_key_version: "encryptionkey@0"
I20260620 11:42:30.251838 7849 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task1JRg7N/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781955746769051-7083-0/minicluster-data/ts-1/wal/instance:
uuid: "926a5e7ef3d648f5b24454af0fe49484"
format_stamp: "Formatted at 2026-06-20 11:42:30 on dist-test-slave-cjc6"
server_key: "0260e705983955de7338fc4c214b71b7"
server_key_iv: "3dab60fd74de29c5d2347493c33d3ec3"
server_key_version: "encryptionkey@0"
I20260620 11:42:30.253162 7849 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260620 11:42:30.254067 7864 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 11:42:30.254257 7849 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260620 11:42:30.254347 7849 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task1JRg7N/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781955746769051-7083-0/minicluster-data/ts-1/data,/tmp/dist-test-task1JRg7N/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781955746769051-7083-0/minicluster-data/ts-1/wal
uuid: "926a5e7ef3d648f5b24454af0fe49484"
format_stamp: "Formatted at 2026-06-20 11:42:30 on dist-test-slave-cjc6"
server_key: "0260e705983955de7338fc4c214b71b7"
server_key_iv: "3dab60fd74de29c5d2347493c33d3ec3"
server_key_version: "encryptionkey@0"
I20260620 11:42:30.254420 7849 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task1JRg7N/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781955746769051-7083-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-task1JRg7N/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781955746769051-7083-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-task1JRg7N/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781955746769051-7083-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 11:42:30.265472 7849 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 11:42:30.267925 7849 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 11:42:30.268085 7849 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 11:42:30.268324 7849 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 11:42:30.268637 7849 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 11:42:30.268672 7849 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 11:42:30.268713 7849 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 11:42:30.268740 7849 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 11:42:30.274654 7849 rpc_server.cc:307] RPC server started. Bound to: 127.6.234.194:45457
I20260620 11:42:30.274715 7977 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.6.234.194:45457 every 8 connection(s)
I20260620 11:42:30.275035 7849 server_base.cc:1166] Dumped server information to /tmp/dist-test-task1JRg7N/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781955746769051-7083-0/minicluster-data/ts-1/data/info.pb
Jun 20 11:42:30 dist-test-slave-cjc6 krb5kdc[7608](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781955750, etypes {rep=17 tkt=17 ses=17}, kudu/127.6.234.194@KRBTEST.COM for kudu/127.6.234.254@KRBTEST.COM
I20260620 11:42:30.281446 7083 external_mini_cluster.cc:1388] Started /tmp/dist-test-task1JRg7N/build/release/bin/kudu as pid 7849
I20260620 11:42:30.281533 7083 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task1JRg7N/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781955746769051-7083-0/minicluster-data/ts-1/wal/instance
I20260620 11:42:30.281746 7083 external_mini_cluster.cc:1428] Setting key 284acd2fb2137ff45912d6660b615b9d
I20260620 11:42:30.284775 7980 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 11:42:30.275793 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.234.254:45361 (local address 127.6.234.194:43213)
0620 11:42:30.275951 (+ 158us) negotiation.cc:107] Waiting for socket to connect
0620 11:42:30.276011 (+ 60us) client_negotiation.cc:174] Beginning negotiation
0620 11:42:30.276910 (+ 899us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 11:42:30.277120 (+ 210us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 11:42:30.277123 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 11:42:30.277372 (+ 249us) client_negotiation.cc:189] Negotiated authn=SASL
0620 11:42:30.277561 (+ 189us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 11:42:30.277565 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 11:42:30.278278 (+ 713us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 11:42:30.278280 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 11:42:30.279296 (+ 1016us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 11:42:30.279299 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 11:42:30.279662 (+ 363us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 11:42:30.279663 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 11:42:30.279777 (+ 114us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 11:42:30.280315 (+ 538us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 11:42:30.280323 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0620 11:42:30.281817 (+ 1494us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 11:42:30.283393 (+ 1576us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 11:42:30.283402 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 11:42:30.283404 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0620 11:42:30.283712 (+ 308us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 11:42:30.283923 (+ 211us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 11:42:30.283923 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 11:42:30.283924 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0620 11:42:30.284031 (+ 107us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 11:42:30.284302 (+ 271us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 11:42:30.284306 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 11:42:30.284416 (+ 110us) client_negotiation.cc:769] Sending connection context
0620 11:42:30.284500 (+ 84us) client_negotiation.cc:240] Negotiation successful
0620 11:42:30.284617 (+ 117us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":118,"thread_start_us":90,"threads_started":1}
I20260620 11:42:30.284777 7699 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 11:42:30.275798 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.6.234.194:43213 (local address 127.6.234.254:45361)
0620 11:42:30.275870 (+ 72us) server_negotiation.cc:206] Beginning negotiation
0620 11:42:30.275872 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 11:42:30.276105 (+ 233us) server_negotiation.cc:407] Connection header received
0620 11:42:30.276997 (+ 892us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 11:42:30.276998 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 11:42:30.277007 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 11:42:30.277067 (+ 60us) server_negotiation.cc:226] Negotiated authn=SASL
0620 11:42:30.277631 (+ 564us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 11:42:30.278200 (+ 569us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 11:42:30.279368 (+ 1168us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 11:42:30.279550 (+ 182us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 11:42:30.279649 (+ 99us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 11:42:30.281900 (+ 2251us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 11:42:30.281903 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 11:42:30.281903 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 11:42:30.281911 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0620 11:42:30.283328 (+ 1417us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 11:42:30.283784 (+ 456us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 11:42:30.283785 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 11:42:30.283786 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 11:42:30.283854 (+ 68us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 11:42:30.284093 (+ 239us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 11:42:30.284094 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 11:42:30.284094 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 11:42:30.284238 (+ 144us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 11:42:30.284309 (+ 71us) server_negotiation.cc:1035] Waiting for connection context
0620 11:42:30.284543 (+ 234us) server_negotiation.cc:299] Negotiation successful
0620 11:42:30.284638 (+ 95us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":39}
I20260620 11:42:30.285204 7978 heartbeater.cc:347] Connected to a master server at 127.6.234.254:45361
I20260620 11:42:30.285298 7978 heartbeater.cc:464] Registering TS with master...
I20260620 11:42:30.285472 7978 heartbeater.cc:511] Master 127.6.234.254:45361 requested a full tablet report, sending...
I20260620 11:42:30.285931 7653 ts_manager.cc:194] Registered new tserver with Master: 926a5e7ef3d648f5b24454af0fe49484 (127.6.234.194:45457)
I20260620 11:42:30.286468 7653 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.6.234.194@KRBTEST.COM'} at 127.6.234.194:43213
WARNING: no policy specified for kudu/127.6.234.195@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.6.234.195@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.6.234.195 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task1JRg7N/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781955746769051-7083-0/krb5kdc/kudu_127.6.234.195.keytab.
Entry for principal kudu/127.6.234.195 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task1JRg7N/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781955746769051-7083-0/krb5kdc/kudu_127.6.234.195.keytab.
I20260620 11:42:30.306538 7083 external_mini_cluster.cc:1326] Running /tmp/dist-test-task1JRg7N/build/release/bin/kudu
/tmp/dist-test-task1JRg7N/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task1JRg7N/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781955746769051-7083-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-task1JRg7N/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781955746769051-7083-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task1JRg7N/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781955746769051-7083-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-task1JRg7N/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781955746769051-7083-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.6.234.195:0
--local_ip_for_outbound_sockets=127.6.234.195
--webserver_interface=127.6.234.195
--webserver_port=0
--tserver_master_addrs=127.6.234.254:45361
--keytab_file=/tmp/dist-test-task1JRg7N/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781955746769051-7083-0/krb5kdc/kudu_127.6.234.195.keytab
--principal=kudu/127.6.234.195
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.6.234.212:40445
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_enforce_access_control with env {KRB5CCNAME=/tmp/dist-test-task1JRg7N/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781955746769051-7083-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task1JRg7N/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781955746769051-7083-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task1JRg7N/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781955746769051-7083-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 11:42:30.387635 7983 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 11:42:30.387883 7983 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 11:42:30.387923 7983 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 11:42:30.389101 7983 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 11:42:30.389169 7983 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 11:42:30.389214 7983 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.6.234.195
I20260620 11:42:30.390765 7983 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.6.234.212:40445
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task1JRg7N/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781955746769051-7083-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-task1JRg7N/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781955746769051-7083-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task1JRg7N/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781955746769051-7083-0/krb5kdc/kudu_127.6.234.195.keytab
--principal=kudu/127.6.234.195
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.6.234.195:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task1JRg7N/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781955746769051-7083-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.6.234.195
--webserver_port=0
--tserver_master_addrs=127.6.234.254:45361
--tserver_enforce_access_control=true
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.7983
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.6.234.195
--log_dir=/tmp/dist-test-task1JRg7N/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781955746769051-7083-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 20 Jun 2026 10:32:37 UTC on ccycloud.quasar-bgyofv.root.comops.site
build id 4457173
I20260620 11:42:30.391048 7983 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 11:42:30.391289 7983 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 11:42:30.396162 7991 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 11:42:30.396183 7988 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 11:42:30.396265 7989 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 11:42:30.396551 7983 server_base.cc:1034] running on GCE node
I20260620 11:42:30.396770 7983 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 11:42:30.397008 7983 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 11:42:30.398186 7983 hybrid_clock.cc:648] HybridClock initialized: now 1781955750398180 us; error 35 us; skew 500 ppm
Jun 20 11:42:30 dist-test-slave-cjc6 krb5kdc[7608](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781955750, etypes {rep=17 tkt=17 ses=17}, kudu/127.6.234.195@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 11:42:30.400693 7983 init.cc:377] Logged in from keytab as kudu/127.6.234.195@KRBTEST.COM (short username kudu)
I20260620 11:42:30.401265 7983 webserver.cc:460] Webserver started at http://127.6.234.195:38983/ using document root <none> and password file <none>
I20260620 11:42:30.401510 7983 fs_manager.cc:359] Metadata directory not provided
I20260620 11:42:30.401579 7983 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 11:42:30.401746 7983 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 11:42:30.402796 7983 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task1JRg7N/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781955746769051-7083-0/minicluster-data/ts-2/data/instance:
uuid: "19d68fbe341a4323beb50880f3fe08d2"
format_stamp: "Formatted at 2026-06-20 11:42:30 on dist-test-slave-cjc6"
server_key: "79a7629ac68b1f86527d0a281f41cf47"
server_key_iv: "6661945558526026aa21d7e3143f197e"
server_key_version: "encryptionkey@0"
I20260620 11:42:30.403167 7983 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task1JRg7N/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781955746769051-7083-0/minicluster-data/ts-2/wal/instance:
uuid: "19d68fbe341a4323beb50880f3fe08d2"
format_stamp: "Formatted at 2026-06-20 11:42:30 on dist-test-slave-cjc6"
server_key: "79a7629ac68b1f86527d0a281f41cf47"
server_key_iv: "6661945558526026aa21d7e3143f197e"
server_key_version: "encryptionkey@0"
I20260620 11:42:30.404465 7983 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.003s
I20260620 11:42:30.405192 7998 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 11:42:30.405372 7983 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.001s
I20260620 11:42:30.405470 7983 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task1JRg7N/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781955746769051-7083-0/minicluster-data/ts-2/data,/tmp/dist-test-task1JRg7N/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781955746769051-7083-0/minicluster-data/ts-2/wal
uuid: "19d68fbe341a4323beb50880f3fe08d2"
format_stamp: "Formatted at 2026-06-20 11:42:30 on dist-test-slave-cjc6"
server_key: "79a7629ac68b1f86527d0a281f41cf47"
server_key_iv: "6661945558526026aa21d7e3143f197e"
server_key_version: "encryptionkey@0"
I20260620 11:42:30.405555 7983 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task1JRg7N/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781955746769051-7083-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-task1JRg7N/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781955746769051-7083-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-task1JRg7N/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781955746769051-7083-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 11:42:30.420338 7983 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 11:42:30.422828 7983 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 11:42:30.423110 7983 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 11:42:30.423297 7983 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 11:42:30.423668 7983 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 11:42:30.423722 7983 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 11:42:30.423764 7983 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 11:42:30.423782 7983 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 11:42:30.430225 7983 rpc_server.cc:307] RPC server started. Bound to: 127.6.234.195:41301
I20260620 11:42:30.430264 8111 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.6.234.195:41301 every 8 connection(s)
I20260620 11:42:30.430603 7983 server_base.cc:1166] Dumped server information to /tmp/dist-test-task1JRg7N/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781955746769051-7083-0/minicluster-data/ts-2/data/info.pb
I20260620 11:42:30.431437 7083 external_mini_cluster.cc:1388] Started /tmp/dist-test-task1JRg7N/build/release/bin/kudu as pid 7983
I20260620 11:42:30.431556 7083 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task1JRg7N/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781955746769051-7083-0/minicluster-data/ts-2/wal/instance
I20260620 11:42:30.431751 7083 external_mini_cluster.cc:1428] Setting key 538d48b0eca135ac78572002356be56d
Jun 20 11:42:30 dist-test-slave-cjc6 krb5kdc[7608](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781955750, etypes {rep=17 tkt=17 ses=17}, kudu/127.6.234.195@KRBTEST.COM for kudu/127.6.234.254@KRBTEST.COM
I20260620 11:42:30.440335 7699 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 11:42:30.431380 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.6.234.195:51287 (local address 127.6.234.254:45361)
0620 11:42:30.431460 (+ 80us) server_negotiation.cc:206] Beginning negotiation
0620 11:42:30.431462 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 11:42:30.431662 (+ 200us) server_negotiation.cc:407] Connection header received
0620 11:42:30.432971 (+ 1309us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 11:42:30.432972 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 11:42:30.432983 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 11:42:30.433033 (+ 50us) server_negotiation.cc:226] Negotiated authn=SASL
0620 11:42:30.433677 (+ 644us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 11:42:30.434252 (+ 575us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 11:42:30.435231 (+ 979us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 11:42:30.435408 (+ 177us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 11:42:30.435462 (+ 54us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 11:42:30.437622 (+ 2160us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 11:42:30.437624 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 11:42:30.437625 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 11:42:30.437631 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0620 11:42:30.439088 (+ 1457us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 11:42:30.439449 (+ 361us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 11:42:30.439450 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 11:42:30.439450 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 11:42:30.439504 (+ 54us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 11:42:30.439750 (+ 246us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 11:42:30.439751 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 11:42:30.439751 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 11:42:30.439866 (+ 115us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 11:42:30.439950 (+ 84us) server_negotiation.cc:1035] Waiting for connection context
0620 11:42:30.440205 (+ 255us) server_negotiation.cc:299] Negotiation successful
0620 11:42:30.440268 (+ 63us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":47}
I20260620 11:42:30.440531 8114 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 11:42:30.431333 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.234.254:45361 (local address 127.6.234.195:51287)
0620 11:42:30.431530 (+ 197us) negotiation.cc:107] Waiting for socket to connect
0620 11:42:30.431575 (+ 45us) client_negotiation.cc:174] Beginning negotiation
0620 11:42:30.432894 (+ 1319us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 11:42:30.433094 (+ 200us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 11:42:30.433097 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 11:42:30.433355 (+ 258us) client_negotiation.cc:189] Negotiated authn=SASL
0620 11:42:30.433572 (+ 217us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 11:42:30.433576 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 11:42:30.434336 (+ 760us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 11:42:30.434337 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 11:42:30.435152 (+ 815us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 11:42:30.435154 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 11:42:30.435480 (+ 326us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 11:42:30.435480 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 11:42:30.435591 (+ 111us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 11:42:30.436135 (+ 544us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 11:42:30.436149 (+ 14us) client_negotiation.cc:656] calling sasl_client_start()
0620 11:42:30.437527 (+ 1378us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 11:42:30.439162 (+ 1635us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 11:42:30.439167 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 11:42:30.439169 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0620 11:42:30.439383 (+ 214us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 11:42:30.439567 (+ 184us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 11:42:30.439568 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 11:42:30.439569 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0620 11:42:30.439673 (+ 104us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 11:42:30.439931 (+ 258us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 11:42:30.439935 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 11:42:30.440066 (+ 131us) client_negotiation.cc:769] Sending connection context
0620 11:42:30.440170 (+ 104us) client_negotiation.cc:240] Negotiation successful
0620 11:42:30.440291 (+ 121us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":137,"thread_start_us":108,"threads_started":1}
I20260620 11:42:30.441058 8112 heartbeater.cc:347] Connected to a master server at 127.6.234.254:45361
I20260620 11:42:30.441149 8112 heartbeater.cc:464] Registering TS with master...
I20260620 11:42:30.441390 8112 heartbeater.cc:511] Master 127.6.234.254:45361 requested a full tablet report, sending...
I20260620 11:42:30.441797 7653 ts_manager.cc:194] Registered new tserver with Master: 19d68fbe341a4323beb50880f3fe08d2 (127.6.234.195:41301)
I20260620 11:42:30.442174 7653 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.6.234.195@KRBTEST.COM'} at 127.6.234.195:51287
I20260620 11:42:30.443606 7083 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260620 11:42:30.451047 7699 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 11:42:30.445083 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53044 (local address 127.6.234.254:45361)
0620 11:42:30.445147 (+ 64us) server_negotiation.cc:206] Beginning negotiation
0620 11:42:30.445149 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 11:42:30.445218 (+ 69us) server_negotiation.cc:407] Connection header received
0620 11:42:30.445293 (+ 75us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 11:42:30.445294 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 11:42:30.445303 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 11:42:30.445357 (+ 54us) server_negotiation.cc:226] Negotiated authn=SASL
0620 11:42:30.445962 (+ 605us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 11:42:30.446427 (+ 465us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 11:42:30.447147 (+ 720us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 11:42:30.447350 (+ 203us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 11:42:30.447432 (+ 82us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 11:42:30.448572 (+ 1140us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 11:42:30.448574 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 11:42:30.448574 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 11:42:30.448581 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0620 11:42:30.449873 (+ 1292us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 11:42:30.450230 (+ 357us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 11:42:30.450231 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 11:42:30.450231 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 11:42:30.450284 (+ 53us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 11:42:30.450488 (+ 204us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 11:42:30.450489 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 11:42:30.450490 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 11:42:30.450641 (+ 151us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 11:42:30.450761 (+ 120us) server_negotiation.cc:1035] Waiting for connection context
0620 11:42:30.450868 (+ 107us) server_negotiation.cc:299] Negotiation successful
0620 11:42:30.450942 (+ 74us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":38}
I20260620 11:42:30.453022 7653 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:53044:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260620 11:42:30.454635 7653 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260620 11:42:30.460737 8123 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 11:42:30.456901 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.234.195:41301 (local address 127.0.0.1:49564)
0620 11:42:30.457040 (+ 139us) negotiation.cc:107] Waiting for socket to connect
0620 11:42:30.457085 (+ 45us) client_negotiation.cc:174] Beginning negotiation
0620 11:42:30.457155 (+ 70us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 11:42:30.457389 (+ 234us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 11:42:30.457392 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 11:42:30.457395 (+ 3us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 11:42:30.457465 (+ 70us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 11:42:30.457466 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 11:42:30.458680 (+ 1214us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 11:42:30.458680 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 11:42:30.459491 (+ 811us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 11:42:30.459494 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 11:42:30.460435 (+ 941us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 11:42:30.460436 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 11:42:30.460515 (+ 79us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 11:42:30.460529 (+ 14us) client_negotiation.cc:769] Sending connection context
0620 11:42:30.460605 (+ 76us) client_negotiation.cc:240] Negotiation successful
0620 11:42:30.460657 (+ 52us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":85,"thread_start_us":47,"threads_started":1}
I20260620 11:42:30.460767 8124 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 11:42:30.457012 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49564 (local address 127.6.234.195:41301)
0620 11:42:30.457194 (+ 182us) server_negotiation.cc:206] Beginning negotiation
0620 11:42:30.457200 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0620 11:42:30.457212 (+ 12us) server_negotiation.cc:407] Connection header received
0620 11:42:30.457254 (+ 42us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 11:42:30.457256 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 11:42:30.457272 (+ 16us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 11:42:30.457362 (+ 90us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 11:42:30.457539 (+ 177us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 11:42:30.458579 (+ 1040us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 11:42:30.459622 (+ 1043us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 11:42:30.460356 (+ 734us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 11:42:30.460430 (+ 74us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 11:42:30.460482 (+ 52us) server_negotiation.cc:1035] Waiting for connection context
0620 11:42:30.460632 (+ 150us) server_negotiation.cc:299] Negotiation successful
0620 11:42:30.460684 (+ 52us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":147,"thread_start_us":93,"threads_started":1}
I20260620 11:42:30.461308 8126 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 11:42:30.457436 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.234.194:45457 (local address 127.0.0.1:43446)
0620 11:42:30.457578 (+ 142us) negotiation.cc:107] Waiting for socket to connect
0620 11:42:30.457587 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0620 11:42:30.457659 (+ 72us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 11:42:30.457925 (+ 266us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 11:42:30.457925 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 11:42:30.457927 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 11:42:30.457964 (+ 37us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 11:42:30.457965 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 11:42:30.459294 (+ 1329us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 11:42:30.459295 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 11:42:30.460228 (+ 933us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 11:42:30.460231 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 11:42:30.461083 (+ 852us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 11:42:30.461084 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 11:42:30.461145 (+ 61us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 11:42:30.461149 (+ 4us) client_negotiation.cc:769] Sending connection context
0620 11:42:30.461204 (+ 55us) client_negotiation.cc:240] Negotiation successful
0620 11:42:30.461241 (+ 37us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":124,"thread_start_us":94,"threads_started":1}
I20260620 11:42:30.461326 8125 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 11:42:30.457271 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43446 (local address 127.6.234.194:45457)
0620 11:42:30.457445 (+ 174us) server_negotiation.cc:206] Beginning negotiation
0620 11:42:30.457450 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 11:42:30.457654 (+ 204us) server_negotiation.cc:407] Connection header received
0620 11:42:30.457746 (+ 92us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 11:42:30.457749 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 11:42:30.457764 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 11:42:30.457837 (+ 73us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 11:42:30.458084 (+ 247us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 11:42:30.459181 (+ 1097us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 11:42:30.460324 (+ 1143us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 11:42:30.461002 (+ 678us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 11:42:30.461083 (+ 81us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 11:42:30.461132 (+ 49us) server_negotiation.cc:1035] Waiting for connection context
0620 11:42:30.461218 (+ 86us) server_negotiation.cc:299] Negotiation successful
0620 11:42:30.461253 (+ 35us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":139,"thread_start_us":87,"threads_started":1}
I20260620 11:42:30.461776 8046 tablet_service.cc:1467] Processing CreateTablet for tablet 162322e65e7149b1a94de6e57d7ae825 (DEFAULT_TABLE table=test-table [id=afd06516b7fa4875ae8e0823ab2bc303]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 11:42:30.462034 8127 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 11:42:30.457705 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.234.193:38839 (local address 127.0.0.1:47136)
0620 11:42:30.457815 (+ 110us) negotiation.cc:107] Waiting for socket to connect
0620 11:42:30.457824 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0620 11:42:30.457864 (+ 40us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 11:42:30.458256 (+ 392us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 11:42:30.458256 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 11:42:30.458259 (+ 3us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 11:42:30.458297 (+ 38us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 11:42:30.458299 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 11:42:30.459557 (+ 1258us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 11:42:30.459558 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 11:42:30.460626 (+ 1068us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 11:42:30.460629 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 11:42:30.461837 (+ 1208us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 11:42:30.461839 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 11:42:30.461896 (+ 57us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 11:42:30.461901 (+ 5us) client_negotiation.cc:769] Sending connection context
0620 11:42:30.461932 (+ 31us) client_negotiation.cc:240] Negotiation successful
0620 11:42:30.461972 (+ 40us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":91,"thread_start_us":72,"threads_started":1}
I20260620 11:42:30.462157 8046 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 162322e65e7149b1a94de6e57d7ae825. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 11:42:30.462038 7912 tablet_service.cc:1467] Processing CreateTablet for tablet 162322e65e7149b1a94de6e57d7ae825 (DEFAULT_TABLE table=test-table [id=afd06516b7fa4875ae8e0823ab2bc303]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 11:42:30.462347 8128 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 11:42:30.457918 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47136 (local address 127.6.234.193:38839)
0620 11:42:30.458093 (+ 175us) server_negotiation.cc:206] Beginning negotiation
0620 11:42:30.458097 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 11:42:30.458106 (+ 9us) server_negotiation.cc:407] Connection header received
0620 11:42:30.458143 (+ 37us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 11:42:30.458145 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 11:42:30.458161 (+ 16us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 11:42:30.458364 (+ 203us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 11:42:30.458441 (+ 77us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 11:42:30.459446 (+ 1005us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 11:42:30.460717 (+ 1271us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 11:42:30.461736 (+ 1019us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 11:42:30.462148 (+ 412us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 11:42:30.462212 (+ 64us) server_negotiation.cc:1035] Waiting for connection context
0620 11:42:30.462237 (+ 25us) server_negotiation.cc:299] Negotiation successful
0620 11:42:30.462276 (+ 39us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":139,"thread_start_us":79,"threads_started":1}
I20260620 11:42:30.462381 7912 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 162322e65e7149b1a94de6e57d7ae825. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 11:42:30.464736 8129 tablet_bootstrap.cc:492] T 162322e65e7149b1a94de6e57d7ae825 P 19d68fbe341a4323beb50880f3fe08d2: Bootstrap starting.
I20260620 11:42:30.465062 8130 tablet_bootstrap.cc:492] T 162322e65e7149b1a94de6e57d7ae825 P 926a5e7ef3d648f5b24454af0fe49484: Bootstrap starting.
I20260620 11:42:30.465458 8129 tablet_bootstrap.cc:654] T 162322e65e7149b1a94de6e57d7ae825 P 19d68fbe341a4323beb50880f3fe08d2: Neither blocks nor log segments found. Creating new log.
I20260620 11:42:30.465768 8130 tablet_bootstrap.cc:654] T 162322e65e7149b1a94de6e57d7ae825 P 926a5e7ef3d648f5b24454af0fe49484: Neither blocks nor log segments found. Creating new log.
I20260620 11:42:30.465788 7778 tablet_service.cc:1467] Processing CreateTablet for tablet 162322e65e7149b1a94de6e57d7ae825 (DEFAULT_TABLE table=test-table [id=afd06516b7fa4875ae8e0823ab2bc303]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 11:42:30.466137 7778 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 162322e65e7149b1a94de6e57d7ae825. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 11:42:30.466150 8129 log.cc:824] T 162322e65e7149b1a94de6e57d7ae825 P 19d68fbe341a4323beb50880f3fe08d2: Log is configured to *not* fsync() on all Append() calls
I20260620 11:42:30.466331 8130 log.cc:824] T 162322e65e7149b1a94de6e57d7ae825 P 926a5e7ef3d648f5b24454af0fe49484: Log is configured to *not* fsync() on all Append() calls
I20260620 11:42:30.466956 8129 tablet_bootstrap.cc:492] T 162322e65e7149b1a94de6e57d7ae825 P 19d68fbe341a4323beb50880f3fe08d2: No bootstrap required, opened a new log
I20260620 11:42:30.467013 8129 ts_tablet_manager.cc:1397] T 162322e65e7149b1a94de6e57d7ae825 P 19d68fbe341a4323beb50880f3fe08d2: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20260620 11:42:30.467231 8130 tablet_bootstrap.cc:492] T 162322e65e7149b1a94de6e57d7ae825 P 926a5e7ef3d648f5b24454af0fe49484: No bootstrap required, opened a new log
I20260620 11:42:30.467353 8130 ts_tablet_manager.cc:1397] T 162322e65e7149b1a94de6e57d7ae825 P 926a5e7ef3d648f5b24454af0fe49484: Time spent bootstrapping tablet: real 0.002s user 0.000s sys 0.002s
I20260620 11:42:30.468278 8129 raft_consensus.cc:348] T 162322e65e7149b1a94de6e57d7ae825 P 19d68fbe341a4323beb50880f3fe08d2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "19d68fbe341a4323beb50880f3fe08d2" member_type: VOTER last_known_addr { host: "127.6.234.195" port: 41301 } } peers { permanent_uuid: "926a5e7ef3d648f5b24454af0fe49484" member_type: VOTER last_known_addr { host: "127.6.234.194" port: 45457 } } peers { permanent_uuid: "6409f1c1cd7948bf8191639fe8679991" member_type: VOTER last_known_addr { host: "127.6.234.193" port: 38839 } }
I20260620 11:42:30.468417 8129 raft_consensus.cc:374] T 162322e65e7149b1a94de6e57d7ae825 P 19d68fbe341a4323beb50880f3fe08d2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 11:42:30.468441 8129 raft_consensus.cc:729] T 162322e65e7149b1a94de6e57d7ae825 P 19d68fbe341a4323beb50880f3fe08d2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 19d68fbe341a4323beb50880f3fe08d2, State: Initialized, Role: FOLLOWER
I20260620 11:42:30.468556 8129 consensus_queue.cc:260] T 162322e65e7149b1a94de6e57d7ae825 P 19d68fbe341a4323beb50880f3fe08d2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "19d68fbe341a4323beb50880f3fe08d2" member_type: VOTER last_known_addr { host: "127.6.234.195" port: 41301 } } peers { permanent_uuid: "926a5e7ef3d648f5b24454af0fe49484" member_type: VOTER last_known_addr { host: "127.6.234.194" port: 45457 } } peers { permanent_uuid: "6409f1c1cd7948bf8191639fe8679991" member_type: VOTER last_known_addr { host: "127.6.234.193" port: 38839 } }
I20260620 11:42:30.468822 8112 heartbeater.cc:503] Master 127.6.234.254:45361 was elected leader, sending a full tablet report...
I20260620 11:42:30.468825 8129 ts_tablet_manager.cc:1428] T 162322e65e7149b1a94de6e57d7ae825 P 19d68fbe341a4323beb50880f3fe08d2: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20260620 11:42:30.469529 8134 tablet_bootstrap.cc:492] T 162322e65e7149b1a94de6e57d7ae825 P 6409f1c1cd7948bf8191639fe8679991: Bootstrap starting.
I20260620 11:42:30.470243 8134 tablet_bootstrap.cc:654] T 162322e65e7149b1a94de6e57d7ae825 P 6409f1c1cd7948bf8191639fe8679991: Neither blocks nor log segments found. Creating new log.
I20260620 11:42:30.470116 8130 raft_consensus.cc:348] T 162322e65e7149b1a94de6e57d7ae825 P 926a5e7ef3d648f5b24454af0fe49484 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "19d68fbe341a4323beb50880f3fe08d2" member_type: VOTER last_known_addr { host: "127.6.234.195" port: 41301 } } peers { permanent_uuid: "926a5e7ef3d648f5b24454af0fe49484" member_type: VOTER last_known_addr { host: "127.6.234.194" port: 45457 } } peers { permanent_uuid: "6409f1c1cd7948bf8191639fe8679991" member_type: VOTER last_known_addr { host: "127.6.234.193" port: 38839 } }
I20260620 11:42:30.470289 8130 raft_consensus.cc:374] T 162322e65e7149b1a94de6e57d7ae825 P 926a5e7ef3d648f5b24454af0fe49484 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 11:42:30.470332 8130 raft_consensus.cc:729] T 162322e65e7149b1a94de6e57d7ae825 P 926a5e7ef3d648f5b24454af0fe49484 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 926a5e7ef3d648f5b24454af0fe49484, State: Initialized, Role: FOLLOWER
I20260620 11:42:30.470445 8130 consensus_queue.cc:260] T 162322e65e7149b1a94de6e57d7ae825 P 926a5e7ef3d648f5b24454af0fe49484 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "19d68fbe341a4323beb50880f3fe08d2" member_type: VOTER last_known_addr { host: "127.6.234.195" port: 41301 } } peers { permanent_uuid: "926a5e7ef3d648f5b24454af0fe49484" member_type: VOTER last_known_addr { host: "127.6.234.194" port: 45457 } } peers { permanent_uuid: "6409f1c1cd7948bf8191639fe8679991" member_type: VOTER last_known_addr { host: "127.6.234.193" port: 38839 } }
I20260620 11:42:30.470732 8130 ts_tablet_manager.cc:1428] T 162322e65e7149b1a94de6e57d7ae825 P 926a5e7ef3d648f5b24454af0fe49484: Time spent starting tablet: real 0.003s user 0.002s sys 0.001s
I20260620 11:42:30.470862 7978 heartbeater.cc:503] Master 127.6.234.254:45361 was elected leader, sending a full tablet report...
I20260620 11:42:30.470865 8134 log.cc:824] T 162322e65e7149b1a94de6e57d7ae825 P 6409f1c1cd7948bf8191639fe8679991: Log is configured to *not* fsync() on all Append() calls
I20260620 11:42:30.471591 8134 tablet_bootstrap.cc:492] T 162322e65e7149b1a94de6e57d7ae825 P 6409f1c1cd7948bf8191639fe8679991: No bootstrap required, opened a new log
I20260620 11:42:30.471671 8134 ts_tablet_manager.cc:1397] T 162322e65e7149b1a94de6e57d7ae825 P 6409f1c1cd7948bf8191639fe8679991: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20260620 11:42:30.473129 8134 raft_consensus.cc:348] T 162322e65e7149b1a94de6e57d7ae825 P 6409f1c1cd7948bf8191639fe8679991 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "19d68fbe341a4323beb50880f3fe08d2" member_type: VOTER last_known_addr { host: "127.6.234.195" port: 41301 } } peers { permanent_uuid: "926a5e7ef3d648f5b24454af0fe49484" member_type: VOTER last_known_addr { host: "127.6.234.194" port: 45457 } } peers { permanent_uuid: "6409f1c1cd7948bf8191639fe8679991" member_type: VOTER last_known_addr { host: "127.6.234.193" port: 38839 } }
I20260620 11:42:30.473272 8134 raft_consensus.cc:374] T 162322e65e7149b1a94de6e57d7ae825 P 6409f1c1cd7948bf8191639fe8679991 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 11:42:30.473317 8134 raft_consensus.cc:729] T 162322e65e7149b1a94de6e57d7ae825 P 6409f1c1cd7948bf8191639fe8679991 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6409f1c1cd7948bf8191639fe8679991, State: Initialized, Role: FOLLOWER
I20260620 11:42:30.473428 8134 consensus_queue.cc:260] T 162322e65e7149b1a94de6e57d7ae825 P 6409f1c1cd7948bf8191639fe8679991 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "19d68fbe341a4323beb50880f3fe08d2" member_type: VOTER last_known_addr { host: "127.6.234.195" port: 41301 } } peers { permanent_uuid: "926a5e7ef3d648f5b24454af0fe49484" member_type: VOTER last_known_addr { host: "127.6.234.194" port: 45457 } } peers { permanent_uuid: "6409f1c1cd7948bf8191639fe8679991" member_type: VOTER last_known_addr { host: "127.6.234.193" port: 38839 } }
I20260620 11:42:30.473659 8134 ts_tablet_manager.cc:1428] T 162322e65e7149b1a94de6e57d7ae825 P 6409f1c1cd7948bf8191639fe8679991: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20260620 11:42:30.473688 7844 heartbeater.cc:503] Master 127.6.234.254:45361 was elected leader, sending a full tablet report...
I20260620 11:42:30.478770 8135 raft_consensus.cc:482] T 162322e65e7149b1a94de6e57d7ae825 P 926a5e7ef3d648f5b24454af0fe49484 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260620 11:42:30.478886 8135 raft_consensus.cc:504] T 162322e65e7149b1a94de6e57d7ae825 P 926a5e7ef3d648f5b24454af0fe49484 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "19d68fbe341a4323beb50880f3fe08d2" member_type: VOTER last_known_addr { host: "127.6.234.195" port: 41301 } } peers { permanent_uuid: "926a5e7ef3d648f5b24454af0fe49484" member_type: VOTER last_known_addr { host: "127.6.234.194" port: 45457 } } peers { permanent_uuid: "6409f1c1cd7948bf8191639fe8679991" member_type: VOTER last_known_addr { host: "127.6.234.193" port: 38839 } }
I20260620 11:42:30.479210 8135 leader_election.cc:288] T 162322e65e7149b1a94de6e57d7ae825 P 926a5e7ef3d648f5b24454af0fe49484 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 19d68fbe341a4323beb50880f3fe08d2 (127.6.234.195:41301), 6409f1c1cd7948bf8191639fe8679991 (127.6.234.193:38839)
I20260620 11:42:30.482306 8124 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 11:42:30.479505 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.6.234.194:56603 (local address 127.6.234.195:41301)
0620 11:42:30.479583 (+ 78us) server_negotiation.cc:206] Beginning negotiation
0620 11:42:30.479585 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 11:42:30.479596 (+ 11us) server_negotiation.cc:407] Connection header received
0620 11:42:30.479657 (+ 61us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 11:42:30.479658 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 11:42:30.479668 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 11:42:30.479724 (+ 56us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 11:42:30.479878 (+ 154us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 11:42:30.480421 (+ 543us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 11:42:30.481488 (+ 1067us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 11:42:30.481949 (+ 461us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 11:42:30.482032 (+ 83us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 11:42:30.482041 (+ 9us) server_negotiation.cc:1035] Waiting for connection context
0620 11:42:30.482203 (+ 162us) server_negotiation.cc:299] Negotiation successful
0620 11:42:30.482238 (+ 35us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":28}
I20260620 11:42:30.482306 7980 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 11:42:30.479426 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.234.195:41301 (local address 127.6.234.194:56603)
0620 11:42:30.479515 (+ 89us) negotiation.cc:107] Waiting for socket to connect
0620 11:42:30.479529 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0620 11:42:30.479592 (+ 63us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 11:42:30.479739 (+ 147us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 11:42:30.479740 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 11:42:30.479742 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 11:42:30.479817 (+ 75us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 11:42:30.479818 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 11:42:30.480515 (+ 697us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 11:42:30.480516 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 11:42:30.481376 (+ 860us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 11:42:30.481379 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 11:42:30.482028 (+ 649us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 11:42:30.482029 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 11:42:30.482111 (+ 82us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 11:42:30.482117 (+ 6us) client_negotiation.cc:769] Sending connection context
0620 11:42:30.482194 (+ 77us) client_negotiation.cc:240] Negotiation successful
0620 11:42:30.482237 (+ 43us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":68,"mutex_wait_us":29}
I20260620 11:42:30.482367 8128 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 11:42:30.479768 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.6.234.194:53287 (local address 127.6.234.193:38839)
0620 11:42:30.479807 (+ 39us) server_negotiation.cc:206] Beginning negotiation
0620 11:42:30.479808 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0620 11:42:30.479815 (+ 7us) server_negotiation.cc:407] Connection header received
0620 11:42:30.479837 (+ 22us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 11:42:30.479838 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 11:42:30.479847 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 11:42:30.479912 (+ 65us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 11:42:30.480098 (+ 186us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 11:42:30.480541 (+ 443us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 11:42:30.481568 (+ 1027us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 11:42:30.482018 (+ 450us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 11:42:30.482101 (+ 83us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 11:42:30.482109 (+ 8us) server_negotiation.cc:1035] Waiting for connection context
0620 11:42:30.482267 (+ 158us) server_negotiation.cc:299] Negotiation successful
0620 11:42:30.482303 (+ 36us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":19}
I20260620 11:42:30.482506 8138 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 11:42:30.479586 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.234.193:38839 (local address 127.6.234.194:53287)
0620 11:42:30.479743 (+ 157us) negotiation.cc:107] Waiting for socket to connect
0620 11:42:30.479752 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0620 11:42:30.479793 (+ 41us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 11:42:30.479932 (+ 139us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 11:42:30.479933 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 11:42:30.479935 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 11:42:30.480024 (+ 89us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 11:42:30.480026 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 11:42:30.480633 (+ 607us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 11:42:30.480634 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 11:42:30.481461 (+ 827us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 11:42:30.481463 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 11:42:30.482101 (+ 638us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 11:42:30.482102 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 11:42:30.482180 (+ 78us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 11:42:30.482184 (+ 4us) client_negotiation.cc:769] Sending connection context
0620 11:42:30.482258 (+ 74us) client_negotiation.cc:240] Negotiation successful
0620 11:42:30.482297 (+ 39us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":141,"thread_start_us":95,"threads_started":1}
I20260620 11:42:30.482669 8066 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "162322e65e7149b1a94de6e57d7ae825" candidate_uuid: "926a5e7ef3d648f5b24454af0fe49484" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "19d68fbe341a4323beb50880f3fe08d2" is_pre_election: true
I20260620 11:42:30.482755 7798 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "162322e65e7149b1a94de6e57d7ae825" candidate_uuid: "926a5e7ef3d648f5b24454af0fe49484" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6409f1c1cd7948bf8191639fe8679991" is_pre_election: true
I20260620 11:42:30.482811 8066 raft_consensus.cc:2454] T 162322e65e7149b1a94de6e57d7ae825 P 19d68fbe341a4323beb50880f3fe08d2 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 926a5e7ef3d648f5b24454af0fe49484 in term 0.
I20260620 11:42:30.482903 7798 raft_consensus.cc:2454] T 162322e65e7149b1a94de6e57d7ae825 P 6409f1c1cd7948bf8191639fe8679991 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 926a5e7ef3d648f5b24454af0fe49484 in term 0.
I20260620 11:42:30.483006 7867 leader_election.cc:302] T 162322e65e7149b1a94de6e57d7ae825 P 926a5e7ef3d648f5b24454af0fe49484 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 19d68fbe341a4323beb50880f3fe08d2, 926a5e7ef3d648f5b24454af0fe49484; no voters:
I20260620 11:42:30.483121 8135 raft_consensus.cc:2781] T 162322e65e7149b1a94de6e57d7ae825 P 926a5e7ef3d648f5b24454af0fe49484 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260620 11:42:30.483191 8135 raft_consensus.cc:482] T 162322e65e7149b1a94de6e57d7ae825 P 926a5e7ef3d648f5b24454af0fe49484 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260620 11:42:30.483208 8135 raft_consensus.cc:3037] T 162322e65e7149b1a94de6e57d7ae825 P 926a5e7ef3d648f5b24454af0fe49484 [term 0 FOLLOWER]: Advancing to term 1
I20260620 11:42:30.483901 8135 raft_consensus.cc:504] T 162322e65e7149b1a94de6e57d7ae825 P 926a5e7ef3d648f5b24454af0fe49484 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "19d68fbe341a4323beb50880f3fe08d2" member_type: VOTER last_known_addr { host: "127.6.234.195" port: 41301 } } peers { permanent_uuid: "926a5e7ef3d648f5b24454af0fe49484" member_type: VOTER last_known_addr { host: "127.6.234.194" port: 45457 } } peers { permanent_uuid: "6409f1c1cd7948bf8191639fe8679991" member_type: VOTER last_known_addr { host: "127.6.234.193" port: 38839 } }
I20260620 11:42:30.484026 8135 leader_election.cc:288] T 162322e65e7149b1a94de6e57d7ae825 P 926a5e7ef3d648f5b24454af0fe49484 [CANDIDATE]: Term 1 election: Requested vote from peers 19d68fbe341a4323beb50880f3fe08d2 (127.6.234.195:41301), 6409f1c1cd7948bf8191639fe8679991 (127.6.234.193:38839)
I20260620 11:42:30.484193 7798 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "162322e65e7149b1a94de6e57d7ae825" candidate_uuid: "926a5e7ef3d648f5b24454af0fe49484" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6409f1c1cd7948bf8191639fe8679991"
I20260620 11:42:30.484202 8066 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "162322e65e7149b1a94de6e57d7ae825" candidate_uuid: "926a5e7ef3d648f5b24454af0fe49484" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "19d68fbe341a4323beb50880f3fe08d2"
I20260620 11:42:30.484275 7798 raft_consensus.cc:3037] T 162322e65e7149b1a94de6e57d7ae825 P 6409f1c1cd7948bf8191639fe8679991 [term 0 FOLLOWER]: Advancing to term 1
I20260620 11:42:30.484274 8066 raft_consensus.cc:3037] T 162322e65e7149b1a94de6e57d7ae825 P 19d68fbe341a4323beb50880f3fe08d2 [term 0 FOLLOWER]: Advancing to term 1
I20260620 11:42:30.485145 7798 raft_consensus.cc:2454] T 162322e65e7149b1a94de6e57d7ae825 P 6409f1c1cd7948bf8191639fe8679991 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 926a5e7ef3d648f5b24454af0fe49484 in term 1.
I20260620 11:42:30.485145 8066 raft_consensus.cc:2454] T 162322e65e7149b1a94de6e57d7ae825 P 19d68fbe341a4323beb50880f3fe08d2 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 926a5e7ef3d648f5b24454af0fe49484 in term 1.
I20260620 11:42:30.485332 7867 leader_election.cc:302] T 162322e65e7149b1a94de6e57d7ae825 P 926a5e7ef3d648f5b24454af0fe49484 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 6409f1c1cd7948bf8191639fe8679991, 926a5e7ef3d648f5b24454af0fe49484; no voters:
I20260620 11:42:30.485409 8135 raft_consensus.cc:2781] T 162322e65e7149b1a94de6e57d7ae825 P 926a5e7ef3d648f5b24454af0fe49484 [term 1 FOLLOWER]: Leader election won for term 1
I20260620 11:42:30.485558 8135 raft_consensus.cc:686] T 162322e65e7149b1a94de6e57d7ae825 P 926a5e7ef3d648f5b24454af0fe49484 [term 1 LEADER]: Becoming Leader. State: Replica: 926a5e7ef3d648f5b24454af0fe49484, State: Running, Role: LEADER
I20260620 11:42:30.485666 8135 consensus_queue.cc:237] T 162322e65e7149b1a94de6e57d7ae825 P 926a5e7ef3d648f5b24454af0fe49484 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "19d68fbe341a4323beb50880f3fe08d2" member_type: VOTER last_known_addr { host: "127.6.234.195" port: 41301 } } peers { permanent_uuid: "926a5e7ef3d648f5b24454af0fe49484" member_type: VOTER last_known_addr { host: "127.6.234.194" port: 45457 } } peers { permanent_uuid: "6409f1c1cd7948bf8191639fe8679991" member_type: VOTER last_known_addr { host: "127.6.234.193" port: 38839 } }
I20260620 11:42:30.486515 7653 catalog_manager.cc:5495] T 162322e65e7149b1a94de6e57d7ae825 P 926a5e7ef3d648f5b24454af0fe49484 reported cstate change: term changed from 0 to 1, leader changed from <none> to 926a5e7ef3d648f5b24454af0fe49484 (127.6.234.194). New cstate: current_term: 1 leader_uuid: "926a5e7ef3d648f5b24454af0fe49484" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "19d68fbe341a4323beb50880f3fe08d2" member_type: VOTER last_known_addr { host: "127.6.234.195" port: 41301 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "926a5e7ef3d648f5b24454af0fe49484" member_type: VOTER last_known_addr { host: "127.6.234.194" port: 45457 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "6409f1c1cd7948bf8191639fe8679991" member_type: VOTER last_known_addr { host: "127.6.234.193" port: 38839 } health_report { overall_health: UNKNOWN } } }
Jun 20 11:42:30 dist-test-slave-cjc6 krb5kdc[7608](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781955749, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.6.234.193@KRBTEST.COM
I20260620 11:42:30.501843 8128 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 11:42:30.494559 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47144 (local address 127.6.234.193:38839)
0620 11:42:30.494622 (+ 63us) server_negotiation.cc:206] Beginning negotiation
0620 11:42:30.494625 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0620 11:42:30.494635 (+ 10us) server_negotiation.cc:407] Connection header received
0620 11:42:30.494683 (+ 48us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 11:42:30.494684 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 11:42:30.494696 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 11:42:30.494773 (+ 77us) server_negotiation.cc:226] Negotiated authn=SASL
0620 11:42:30.495328 (+ 555us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 11:42:30.495795 (+ 467us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 11:42:30.496510 (+ 715us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 11:42:30.496704 (+ 194us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 11:42:30.496779 (+ 75us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 11:42:30.499287 (+ 2508us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 11:42:30.499296 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 11:42:30.499296 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 11:42:30.499303 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0620 11:42:30.500768 (+ 1465us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 11:42:30.501128 (+ 360us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 11:42:30.501131 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 11:42:30.501132 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 11:42:30.501184 (+ 52us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 11:42:30.501349 (+ 165us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 11:42:30.501350 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 11:42:30.501350 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 11:42:30.501479 (+ 129us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 11:42:30.501581 (+ 102us) server_negotiation.cc:1035] Waiting for connection context
0620 11:42:30.501638 (+ 57us) server_negotiation.cc:299] Negotiation successful
0620 11:42:30.501759 (+ 121us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":40}
Jun 20 11:42:30 dist-test-slave-cjc6 krb5kdc[7608](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781955750, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-user@KRBTEST.COM:
Jun 20 11:42:30 dist-test-slave-cjc6 krb5kdc[7608](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781955750, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for kudu/127.6.234.193@KRBTEST.COM
I20260620 11:42:30.524837 8128 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 11:42:30.517836 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47148 (local address 127.6.234.193:38839)
0620 11:42:30.517912 (+ 76us) server_negotiation.cc:206] Beginning negotiation
0620 11:42:30.517914 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 11:42:30.517948 (+ 34us) server_negotiation.cc:407] Connection header received
0620 11:42:30.518032 (+ 84us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 11:42:30.518033 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 11:42:30.518042 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 11:42:30.518098 (+ 56us) server_negotiation.cc:226] Negotiated authn=SASL
0620 11:42:30.518635 (+ 537us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 11:42:30.519120 (+ 485us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 11:42:30.519854 (+ 734us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 11:42:30.520027 (+ 173us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 11:42:30.520083 (+ 56us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 11:42:30.522311 (+ 2228us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 11:42:30.522312 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 11:42:30.522313 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 11:42:30.522319 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0620 11:42:30.523883 (+ 1564us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 11:42:30.524183 (+ 300us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 11:42:30.524183 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 11:42:30.524184 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 11:42:30.524232 (+ 48us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 11:42:30.524387 (+ 155us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 11:42:30.524388 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 11:42:30.524388 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 11:42:30.524483 (+ 95us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 11:42:30.524556 (+ 73us) server_negotiation.cc:1035] Waiting for connection context
0620 11:42:30.524687 (+ 131us) server_negotiation.cc:299] Negotiation successful
0620 11:42:30.524744 (+ 57us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":52}
W20260620 11:42:30.525628 7979 tablet.cc:2369] T 162322e65e7149b1a94de6e57d7ae825 P 926a5e7ef3d648f5b24454af0fe49484: Can't schedule compaction. Clean time has not been advanced past its initial value.
Jun 20 11:42:30 dist-test-slave-cjc6 krb5kdc[7608](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781955750, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
Jun 20 11:42:30 dist-test-slave-cjc6 krb5kdc[7608](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781955750, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.6.234.193@KRBTEST.COM
I20260620 11:42:30.547817 8128 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 11:42:30.540194 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47164 (local address 127.6.234.193:38839)
0620 11:42:30.540265 (+ 71us) server_negotiation.cc:206] Beginning negotiation
0620 11:42:30.540268 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0620 11:42:30.540370 (+ 102us) server_negotiation.cc:407] Connection header received
0620 11:42:30.540447 (+ 77us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 11:42:30.540448 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 11:42:30.540464 (+ 16us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 11:42:30.540529 (+ 65us) server_negotiation.cc:226] Negotiated authn=SASL
0620 11:42:30.541096 (+ 567us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 11:42:30.541589 (+ 493us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 11:42:30.542426 (+ 837us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 11:42:30.542693 (+ 267us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 11:42:30.542814 (+ 121us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 11:42:30.545183 (+ 2369us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 11:42:30.545184 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 11:42:30.545185 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 11:42:30.545192 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0620 11:42:30.546669 (+ 1477us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 11:42:30.547036 (+ 367us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 11:42:30.547038 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 11:42:30.547038 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 11:42:30.547095 (+ 57us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 11:42:30.547293 (+ 198us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 11:42:30.547294 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 11:42:30.547294 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 11:42:30.547401 (+ 107us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 11:42:30.547500 (+ 99us) server_negotiation.cc:1035] Waiting for connection context
0620 11:42:30.547632 (+ 132us) server_negotiation.cc:299] Negotiation successful
0620 11:42:30.547711 (+ 79us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":41}
W20260620 11:42:30.548350 7758 tablet_service.cc:2908] Rejecting scan request for tablet 162322e65e7149b1a94de6e57d7ae825: Uninitialized: clean time has not yet been initialized
I20260620 11:42:30.549214 7083 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task1JRg7N/build/release/bin/kudu with pid 7715
I20260620 11:42:30.554174 7083 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task1JRg7N/build/release/bin/kudu with pid 7849
I20260620 11:42:30.558769 7083 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task1JRg7N/build/release/bin/kudu with pid 7983
I20260620 11:42:30.563370 7083 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task1JRg7N/build/release/bin/kudu with pid 7621
2026-06-20T11:42:30Z chronyd exiting
[ OK ] SecurityITest.TestAuthorizationOnChecksum (1944 ms)
[ RUN ] SecurityITest.SmokeTestAsAuthorizedUser
Loading random data
Initializing database '/tmp/dist-test-task1JRg7N/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781955746769051-7083-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 20 11:42:30 dist-test-slave-cjc6 krb5kdc[8153](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 20 11:42:30 dist-test-slave-cjc6 krb5kdc[8153](info): set up 2 sockets
Jun 20 11:42:30 dist-test-slave-cjc6 krb5kdc[8153](info): commencing operation
krb5kdc: starting...
W20260620 11:42:31.597241 7083 mini_kdc.cc:121] Time spent starting KDC: real 1.013s user 0.000s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task1JRg7N/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781955746769051-7083-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task1JRg7N/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781955746769051-7083-0/krb5kdc/test-user.keytab.
Jun 20 11:42:31 dist-test-slave-cjc6 krb5kdc[8153](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781955751, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-20T11:42:31Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-20T11:42:31Z Disabled control of system clock
WARNING: no policy specified for kudu/127.6.234.254@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.6.234.254@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.6.234.254 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task1JRg7N/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781955746769051-7083-0/krb5kdc/kudu_127.6.234.254.keytab.
Entry for principal kudu/127.6.234.254 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task1JRg7N/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781955746769051-7083-0/krb5kdc/kudu_127.6.234.254.keytab.
I20260620 11:42:31.714712 7083 external_mini_cluster.cc:1326] Running /tmp/dist-test-task1JRg7N/build/release/bin/kudu
/tmp/dist-test-task1JRg7N/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task1JRg7N/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781955746769051-7083-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task1JRg7N/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781955746769051-7083-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task1JRg7N/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781955746769051-7083-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task1JRg7N/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781955746769051-7083-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.6.234.254:43185
--webserver_interface=127.6.234.254
--webserver_port=0
--keytab_file=/tmp/dist-test-task1JRg7N/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781955746769051-7083-0/krb5kdc/kudu_127.6.234.254.keytab
--principal=kudu/127.6.234.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.6.234.212:42185
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.6.234.254:43185
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--txn_manager_enabled=true with env {KRB5CCNAME=/tmp/dist-test-task1JRg7N/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781955746769051-7083-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task1JRg7N/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781955746769051-7083-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task1JRg7N/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781955746769051-7083-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 11:42:31.793483 8166 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 11:42:31.793788 8166 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 11:42:31.793815 8166 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 11:42:31.794986 8166 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 11:42:31.795027 8166 flags.cc:425] Enabled experimental flag: --txn_manager_enabled=true
W20260620 11:42:31.795040 8166 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 11:42:31.795055 8166 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 11:42:31.795068 8166 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 11:42:31.795125 8166 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 11:42:31.796581 8166 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.6.234.212:42185
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task1JRg7N/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781955746769051-7083-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task1JRg7N/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781955746769051-7083-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.6.234.254:43185
--txn_manager_enabled=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task1JRg7N/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781955746769051-7083-0/krb5kdc/kudu_127.6.234.254.keytab
--principal=kudu/127.6.234.254
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.6.234.254:43185
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task1JRg7N/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781955746769051-7083-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.6.234.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.8166
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task1JRg7N/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781955746769051-7083-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 20 Jun 2026 10:32:37 UTC on ccycloud.quasar-bgyofv.root.comops.site
build id 4457173
I20260620 11:42:31.796835 8166 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 11:42:31.797073 8166 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 11:42:31.801967 8174 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 11:42:31.801946 8172 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 11:42:31.801924 8171 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 11:42:31.802181 8166 server_base.cc:1034] running on GCE node
I20260620 11:42:31.802412 8166 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 11:42:31.802692 8166 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 11:42:31.803875 8166 hybrid_clock.cc:648] HybridClock initialized: now 1781955751803868 us; error 36 us; skew 500 ppm
Jun 20 11:42:31 dist-test-slave-cjc6 krb5kdc[8153](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781955751, etypes {rep=17 tkt=17 ses=17}, kudu/127.6.234.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 11:42:31.806397 8166 init.cc:377] Logged in from keytab as kudu/127.6.234.254@KRBTEST.COM (short username kudu)
I20260620 11:42:31.806882 8166 webserver.cc:460] Webserver started at http://127.6.234.254:34093/ using document root <none> and password file <none>
I20260620 11:42:31.807118 8166 fs_manager.cc:359] Metadata directory not provided
I20260620 11:42:31.807185 8166 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 11:42:31.807337 8166 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 11:42:31.808379 8166 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task1JRg7N/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781955746769051-7083-0/minicluster-data/master-0/data/instance:
uuid: "a56f5849e11d410fa2127cc9991f437c"
format_stamp: "Formatted at 2026-06-20 11:42:31 on dist-test-slave-cjc6"
server_key: "5c282e72a1f8461a87d7a236bdb26e43"
server_key_iv: "69e2ffb87dbd31eefcdf4d16c7459ee5"
server_key_version: "encryptionkey@0"
I20260620 11:42:31.808744 8166 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task1JRg7N/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781955746769051-7083-0/minicluster-data/master-0/wal/instance:
uuid: "a56f5849e11d410fa2127cc9991f437c"
format_stamp: "Formatted at 2026-06-20 11:42:31 on dist-test-slave-cjc6"
server_key: "5c282e72a1f8461a87d7a236bdb26e43"
server_key_iv: "69e2ffb87dbd31eefcdf4d16c7459ee5"
server_key_version: "encryptionkey@0"
I20260620 11:42:31.810125 8166 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260620 11:42:31.810814 8181 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 11:42:31.811002 8166 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260620 11:42:31.811106 8166 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task1JRg7N/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781955746769051-7083-0/minicluster-data/master-0/data,/tmp/dist-test-task1JRg7N/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781955746769051-7083-0/minicluster-data/master-0/wal
uuid: "a56f5849e11d410fa2127cc9991f437c"
format_stamp: "Formatted at 2026-06-20 11:42:31 on dist-test-slave-cjc6"
server_key: "5c282e72a1f8461a87d7a236bdb26e43"
server_key_iv: "69e2ffb87dbd31eefcdf4d16c7459ee5"
server_key_version: "encryptionkey@0"
I20260620 11:42:31.811178 8166 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task1JRg7N/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781955746769051-7083-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task1JRg7N/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781955746769051-7083-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task1JRg7N/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781955746769051-7083-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 11:42:31.827476 8166 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 11:42:31.829841 8166 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 11:42:31.830005 8166 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 11:42:31.834446 8166 rpc_server.cc:307] RPC server started. Bound to: 127.6.234.254:43185
I20260620 11:42:31.834519 8243 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.6.234.254:43185 every 8 connection(s)
I20260620 11:42:31.834863 8166 server_base.cc:1166] Dumped server information to /tmp/dist-test-task1JRg7N/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781955746769051-7083-0/minicluster-data/master-0/data/info.pb
I20260620 11:42:31.835551 8244 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 11:42:31.837754 8244 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P a56f5849e11d410fa2127cc9991f437c: Bootstrap starting.
I20260620 11:42:31.838330 8244 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P a56f5849e11d410fa2127cc9991f437c: Neither blocks nor log segments found. Creating new log.
I20260620 11:42:31.838564 8244 log.cc:824] T 00000000000000000000000000000000 P a56f5849e11d410fa2127cc9991f437c: Log is configured to *not* fsync() on all Append() calls
I20260620 11:42:31.839004 7083 external_mini_cluster.cc:1388] Started /tmp/dist-test-task1JRg7N/build/release/bin/kudu as pid 8166
I20260620 11:42:31.839098 7083 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task1JRg7N/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781955746769051-7083-0/minicluster-data/master-0/wal/instance
I20260620 11:42:31.839267 7083 external_mini_cluster.cc:1428] Setting key 760204588bd26c30adfd881c97984469
I20260620 11:42:31.839288 8244 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P a56f5849e11d410fa2127cc9991f437c: No bootstrap required, opened a new log
I20260620 11:42:31.842141 8244 raft_consensus.cc:348] T 00000000000000000000000000000000 P a56f5849e11d410fa2127cc9991f437c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a56f5849e11d410fa2127cc9991f437c" member_type: VOTER last_known_addr { host: "127.6.234.254" port: 43185 } }
I20260620 11:42:31.842274 8244 raft_consensus.cc:374] T 00000000000000000000000000000000 P a56f5849e11d410fa2127cc9991f437c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 11:42:31.842298 8244 raft_consensus.cc:729] T 00000000000000000000000000000000 P a56f5849e11d410fa2127cc9991f437c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a56f5849e11d410fa2127cc9991f437c, State: Initialized, Role: FOLLOWER
I20260620 11:42:31.842437 8244 consensus_queue.cc:260] T 00000000000000000000000000000000 P a56f5849e11d410fa2127cc9991f437c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a56f5849e11d410fa2127cc9991f437c" member_type: VOTER last_known_addr { host: "127.6.234.254" port: 43185 } }
I20260620 11:42:31.842507 8244 raft_consensus.cc:388] T 00000000000000000000000000000000 P a56f5849e11d410fa2127cc9991f437c [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260620 11:42:31.842549 8244 raft_consensus.cc:482] T 00000000000000000000000000000000 P a56f5849e11d410fa2127cc9991f437c [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260620 11:42:31.842597 8244 raft_consensus.cc:3037] T 00000000000000000000000000000000 P a56f5849e11d410fa2127cc9991f437c [term 0 FOLLOWER]: Advancing to term 1
I20260620 11:42:31.843146 8244 raft_consensus.cc:504] T 00000000000000000000000000000000 P a56f5849e11d410fa2127cc9991f437c [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a56f5849e11d410fa2127cc9991f437c" member_type: VOTER last_known_addr { host: "127.6.234.254" port: 43185 } }
I20260620 11:42:31.843266 8244 leader_election.cc:302] T 00000000000000000000000000000000 P a56f5849e11d410fa2127cc9991f437c [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: a56f5849e11d410fa2127cc9991f437c; no voters:
I20260620 11:42:31.843449 8244 leader_election.cc:288] T 00000000000000000000000000000000 P a56f5849e11d410fa2127cc9991f437c [CANDIDATE]: Term 1 election: Requested vote from peers
I20260620 11:42:31.843477 8249 raft_consensus.cc:2781] T 00000000000000000000000000000000 P a56f5849e11d410fa2127cc9991f437c [term 1 FOLLOWER]: Leader election won for term 1
I20260620 11:42:31.843585 8249 raft_consensus.cc:686] T 00000000000000000000000000000000 P a56f5849e11d410fa2127cc9991f437c [term 1 LEADER]: Becoming Leader. State: Replica: a56f5849e11d410fa2127cc9991f437c, State: Running, Role: LEADER
I20260620 11:42:31.843667 8249 consensus_queue.cc:237] T 00000000000000000000000000000000 P a56f5849e11d410fa2127cc9991f437c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a56f5849e11d410fa2127cc9991f437c" member_type: VOTER last_known_addr { host: "127.6.234.254" port: 43185 } }
I20260620 11:42:31.843706 8244 sys_catalog.cc:564] T 00000000000000000000000000000000 P a56f5849e11d410fa2127cc9991f437c [sys.catalog]: configured and running, proceeding with master startup.
I20260620 11:42:31.843966 8250 sys_catalog.cc:455] T 00000000000000000000000000000000 P a56f5849e11d410fa2127cc9991f437c [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "a56f5849e11d410fa2127cc9991f437c" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a56f5849e11d410fa2127cc9991f437c" member_type: VOTER last_known_addr { host: "127.6.234.254" port: 43185 } } }
I20260620 11:42:31.844228 8250 sys_catalog.cc:458] T 00000000000000000000000000000000 P a56f5849e11d410fa2127cc9991f437c [sys.catalog]: This master's current role is: LEADER
I20260620 11:42:31.844153 8251 sys_catalog.cc:455] T 00000000000000000000000000000000 P a56f5849e11d410fa2127cc9991f437c [sys.catalog]: SysCatalogTable state changed. Reason: New leader a56f5849e11d410fa2127cc9991f437c. Latest consensus state: current_term: 1 leader_uuid: "a56f5849e11d410fa2127cc9991f437c" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a56f5849e11d410fa2127cc9991f437c" member_type: VOTER last_known_addr { host: "127.6.234.254" port: 43185 } } }
I20260620 11:42:31.844506 8251 sys_catalog.cc:458] T 00000000000000000000000000000000 P a56f5849e11d410fa2127cc9991f437c [sys.catalog]: This master's current role is: LEADER
I20260620 11:42:31.844998 8258 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260620 11:42:31.845458 8258 catalog_manager.cc:1484] Initializing Kudu cluster ID...
Jun 20 11:42:31 dist-test-slave-cjc6 krb5kdc[8153](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781955751, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.6.234.254@KRBTEST.COM
I20260620 11:42:31.847203 8258 catalog_manager.cc:1347] Generated new cluster ID: 817679246b2c41ff9d87cdfd847a17dd
I20260620 11:42:31.847257 8258 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260620 11:42:31.850836 8248 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 11:42:31.839833 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46852 (local address 127.6.234.254:43185)
0620 11:42:31.840008 (+ 175us) server_negotiation.cc:206] Beginning negotiation
0620 11:42:31.840016 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0620 11:42:31.840050 (+ 34us) server_negotiation.cc:407] Connection header received
0620 11:42:31.840570 (+ 520us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 11:42:31.840576 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 11:42:31.840621 (+ 45us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 11:42:31.840730 (+ 109us) server_negotiation.cc:226] Negotiated authn=SASL
0620 11:42:31.841570 (+ 840us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 11:42:31.842628 (+ 1058us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 11:42:31.843387 (+ 759us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 11:42:31.844017 (+ 630us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 11:42:31.844817 (+ 800us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 11:42:31.847024 (+ 2207us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 11:42:31.847028 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 11:42:31.847028 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 11:42:31.847040 (+ 12us) server_negotiation.cc:922] calling sasl_server_start()
0620 11:42:31.848781 (+ 1741us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 11:42:31.849172 (+ 391us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 11:42:31.849175 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 11:42:31.849175 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 11:42:31.849265 (+ 90us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 11:42:31.849474 (+ 209us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 11:42:31.849475 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 11:42:31.849475 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 11:42:31.849755 (+ 280us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 11:42:31.850276 (+ 521us) server_negotiation.cc:1035] Waiting for connection context
0620 11:42:31.850522 (+ 246us) server_negotiation.cc:299] Negotiation successful
0620 11:42:31.850657 (+ 135us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":128,"thread_start_us":101,"threads_started":1}
I20260620 11:42:31.887502 8258 catalog_manager.cc:1370] Generated new certificate authority record
I20260620 11:42:31.888245 8258 catalog_manager.cc:1504] Loading token signing keys...
I20260620 11:42:31.891054 8258 catalog_manager.cc:5868] T 00000000000000000000000000000000 P a56f5849e11d410fa2127cc9991f437c: Generated new TSK 0
I20260620 11:42:31.891228 8258 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.6.234.193@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.6.234.193@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.6.234.193 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task1JRg7N/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781955746769051-7083-0/krb5kdc/kudu_127.6.234.193.keytab.
Entry for principal kudu/127.6.234.193 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task1JRg7N/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781955746769051-7083-0/krb5kdc/kudu_127.6.234.193.keytab.
I20260620 11:42:31.922025 7083 external_mini_cluster.cc:1326] Running /tmp/dist-test-task1JRg7N/build/release/bin/kudu
/tmp/dist-test-task1JRg7N/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task1JRg7N/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781955746769051-7083-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-task1JRg7N/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781955746769051-7083-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task1JRg7N/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781955746769051-7083-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-task1JRg7N/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781955746769051-7083-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.6.234.193:0
--local_ip_for_outbound_sockets=127.6.234.193
--webserver_interface=127.6.234.193
--webserver_port=0
--tserver_master_addrs=127.6.234.254:43185
--keytab_file=/tmp/dist-test-task1JRg7N/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781955746769051-7083-0/krb5kdc/kudu_127.6.234.193.keytab
--principal=kudu/127.6.234.193
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.6.234.212:42185
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true with env {KRB5CCNAME=/tmp/dist-test-task1JRg7N/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781955746769051-7083-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task1JRg7N/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781955746769051-7083-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task1JRg7N/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781955746769051-7083-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 11:42:32.003362 8270 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 11:42:32.003618 8270 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 11:42:32.003664 8270 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 11:42:32.004997 8270 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 11:42:32.005069 8270 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 11:42:32.005110 8270 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260620 11:42:32.005174 8270 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.6.234.193
I20260620 11:42:32.006878 8270 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.6.234.212:42185
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task1JRg7N/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781955746769051-7083-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-task1JRg7N/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781955746769051-7083-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task1JRg7N/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781955746769051-7083-0/krb5kdc/kudu_127.6.234.193.keytab
--principal=kudu/127.6.234.193
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.6.234.193:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task1JRg7N/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781955746769051-7083-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.6.234.193
--webserver_port=0
--enable_txn_system_client_init=true
--tserver_master_addrs=127.6.234.254:43185
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.8270
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.6.234.193
--log_dir=/tmp/dist-test-task1JRg7N/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781955746769051-7083-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 20 Jun 2026 10:32:37 UTC on ccycloud.quasar-bgyofv.root.comops.site
build id 4457173
I20260620 11:42:32.007167 8270 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 11:42:32.007413 8270 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 11:42:32.012653 8276 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 11:42:32.012670 8275 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 11:42:32.012975 8270 server_base.cc:1034] running on GCE node
W20260620 11:42:32.013062 8278 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 11:42:32.013258 8270 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 11:42:32.013470 8270 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 11:42:32.014604 8270 hybrid_clock.cc:648] HybridClock initialized: now 1781955752014581 us; error 37 us; skew 500 ppm
Jun 20 11:42:32 dist-test-slave-cjc6 krb5kdc[8153](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781955752, etypes {rep=17 tkt=17 ses=17}, kudu/127.6.234.193@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 11:42:32.017343 8270 init.cc:377] Logged in from keytab as kudu/127.6.234.193@KRBTEST.COM (short username kudu)
I20260620 11:42:32.017889 8270 webserver.cc:460] Webserver started at http://127.6.234.193:42791/ using document root <none> and password file <none>
I20260620 11:42:32.018115 8270 fs_manager.cc:359] Metadata directory not provided
I20260620 11:42:32.018180 8270 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 11:42:32.018329 8270 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 11:42:32.019375 8270 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task1JRg7N/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781955746769051-7083-0/minicluster-data/ts-0/data/instance:
uuid: "df86ed971c13424aba5fe002377e02e8"
format_stamp: "Formatted at 2026-06-20 11:42:32 on dist-test-slave-cjc6"
server_key: "39e103635b3e5046259f3c0cc77bc758"
server_key_iv: "192a6ae608a316cc52f2e1f6e6d48c03"
server_key_version: "encryptionkey@0"
I20260620 11:42:32.019740 8270 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task1JRg7N/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781955746769051-7083-0/minicluster-data/ts-0/wal/instance:
uuid: "df86ed971c13424aba5fe002377e02e8"
format_stamp: "Formatted at 2026-06-20 11:42:32 on dist-test-slave-cjc6"
server_key: "39e103635b3e5046259f3c0cc77bc758"
server_key_iv: "192a6ae608a316cc52f2e1f6e6d48c03"
server_key_version: "encryptionkey@0"
I20260620 11:42:32.021029 8270 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.003s
I20260620 11:42:32.021803 8285 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 11:42:32.021991 8270 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.001s
I20260620 11:42:32.022089 8270 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task1JRg7N/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781955746769051-7083-0/minicluster-data/ts-0/data,/tmp/dist-test-task1JRg7N/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781955746769051-7083-0/minicluster-data/ts-0/wal
uuid: "df86ed971c13424aba5fe002377e02e8"
format_stamp: "Formatted at 2026-06-20 11:42:32 on dist-test-slave-cjc6"
server_key: "39e103635b3e5046259f3c0cc77bc758"
server_key_iv: "192a6ae608a316cc52f2e1f6e6d48c03"
server_key_version: "encryptionkey@0"
I20260620 11:42:32.022169 8270 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task1JRg7N/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781955746769051-7083-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-task1JRg7N/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781955746769051-7083-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-task1JRg7N/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781955746769051-7083-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 11:42:32.032294 8270 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 11:42:32.034293 8270 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 11:42:32.034456 8270 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 11:42:32.034951 8270 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 11:42:32.035013 8270 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 11:42:32.035063 8270 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 11:42:32.035105 8270 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Jun 20 11:42:32 dist-test-slave-cjc6 krb5kdc[8153](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781955752, etypes {rep=17 tkt=17 ses=17}, kudu/127.6.234.193@KRBTEST.COM for kudu/127.6.234.254@KRBTEST.COM
I20260620 11:42:32.042723 8270 rpc_server.cc:307] RPC server started. Bound to: 127.6.234.193:42909
I20260620 11:42:32.043100 8400 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.6.234.193:42909 every 8 connection(s)
I20260620 11:42:32.043177 8270 server_base.cc:1166] Dumped server information to /tmp/dist-test-task1JRg7N/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781955746769051-7083-0/minicluster-data/ts-0/data/info.pb
I20260620 11:42:32.045567 8248 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 11:42:32.035563 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.6.234.193:33887 (local address 127.6.234.254:43185)
0620 11:42:32.035640 (+ 77us) server_negotiation.cc:206] Beginning negotiation
0620 11:42:32.035643 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0620 11:42:32.035873 (+ 230us) server_negotiation.cc:407] Connection header received
0620 11:42:32.036687 (+ 814us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 11:42:32.036688 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 11:42:32.036703 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 11:42:32.036796 (+ 93us) server_negotiation.cc:226] Negotiated authn=SASL
0620 11:42:32.037766 (+ 970us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 11:42:32.038353 (+ 587us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 11:42:32.039393 (+ 1040us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 11:42:32.039632 (+ 239us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 11:42:32.039714 (+ 82us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 11:42:32.042370 (+ 2656us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 11:42:32.042371 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 11:42:32.042371 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 11:42:32.042378 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0620 11:42:32.044084 (+ 1706us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 11:42:32.044558 (+ 474us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 11:42:32.044560 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 11:42:32.044560 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 11:42:32.044617 (+ 57us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 11:42:32.044882 (+ 265us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 11:42:32.044883 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 11:42:32.044883 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 11:42:32.045001 (+ 118us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 11:42:32.045084 (+ 83us) server_negotiation.cc:1035] Waiting for connection context
0620 11:42:32.045376 (+ 292us) server_negotiation.cc:299] Negotiation successful
0620 11:42:32.045474 (+ 98us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":48}
I20260620 11:42:32.045735 8294 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 11:42:32.035525 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.234.254:43185 (local address 127.6.234.193:33887)
0620 11:42:32.035730 (+ 205us) negotiation.cc:107] Waiting for socket to connect
0620 11:42:32.035781 (+ 51us) client_negotiation.cc:174] Beginning negotiation
0620 11:42:32.036511 (+ 730us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 11:42:32.036904 (+ 393us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 11:42:32.036909 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 11:42:32.037316 (+ 407us) client_negotiation.cc:189] Negotiated authn=SASL
0620 11:42:32.037626 (+ 310us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 11:42:32.037634 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 11:42:32.038460 (+ 826us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 11:42:32.038462 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 11:42:32.039282 (+ 820us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 11:42:32.039284 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 11:42:32.039738 (+ 454us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 11:42:32.039739 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 11:42:32.039937 (+ 198us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 11:42:32.040463 (+ 526us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 11:42:32.040477 (+ 14us) client_negotiation.cc:656] calling sasl_client_start()
0620 11:42:32.042265 (+ 1788us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 11:42:32.044188 (+ 1923us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 11:42:32.044195 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 11:42:32.044197 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0620 11:42:32.044464 (+ 267us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 11:42:32.044690 (+ 226us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 11:42:32.044691 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 11:42:32.044692 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0620 11:42:32.044804 (+ 112us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 11:42:32.045075 (+ 271us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 11:42:32.045082 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 11:42:32.045221 (+ 139us) client_negotiation.cc:769] Sending connection context
0620 11:42:32.045338 (+ 117us) client_negotiation.cc:240] Negotiation successful
0620 11:42:32.045488 (+ 150us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":157,"thread_start_us":98,"threads_started":1}
I20260620 11:42:32.046274 8401 heartbeater.cc:347] Connected to a master server at 127.6.234.254:43185
I20260620 11:42:32.046352 8401 heartbeater.cc:464] Registering TS with master...
I20260620 11:42:32.046525 8401 heartbeater.cc:511] Master 127.6.234.254:43185 requested a full tablet report, sending...
I20260620 11:42:32.046949 7083 external_mini_cluster.cc:1388] Started /tmp/dist-test-task1JRg7N/build/release/bin/kudu as pid 8270
I20260620 11:42:32.047071 7083 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task1JRg7N/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781955746769051-7083-0/minicluster-data/ts-0/wal/instance
I20260620 11:42:32.047123 8197 ts_manager.cc:194] Registered new tserver with Master: df86ed971c13424aba5fe002377e02e8 (127.6.234.193:42909)
I20260620 11:42:32.047260 7083 external_mini_cluster.cc:1428] Setting key 13cb294971147a6c0fb51626ed51ed72
I20260620 11:42:32.048062 8197 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.6.234.193@KRBTEST.COM'} at 127.6.234.193:33887
I20260620 11:42:32.053481 8407 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 11:42:32.047482 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.234.254:43185 (local address 127.6.234.193:52603)
0620 11:42:32.047629 (+ 147us) negotiation.cc:107] Waiting for socket to connect
0620 11:42:32.047639 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0620 11:42:32.047699 (+ 60us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 11:42:32.047811 (+ 112us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 11:42:32.047812 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 11:42:32.047993 (+ 181us) client_negotiation.cc:189] Negotiated authn=SASL
0620 11:42:32.048057 (+ 64us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 11:42:32.048059 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 11:42:32.048964 (+ 905us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 11:42:32.048965 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 11:42:32.049871 (+ 906us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 11:42:32.049874 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 11:42:32.050305 (+ 431us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 11:42:32.050306 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 11:42:32.050363 (+ 57us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 11:42:32.050791 (+ 428us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 11:42:32.050796 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0620 11:42:32.051069 (+ 273us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 11:42:32.052480 (+ 1411us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 11:42:32.052481 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 11:42:32.052481 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0620 11:42:32.052653 (+ 172us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 11:42:32.052860 (+ 207us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 11:42:32.052860 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 11:42:32.052861 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0620 11:42:32.052917 (+ 56us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 11:42:32.053159 (+ 242us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 11:42:32.053159 (+ 0us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 11:42:32.053204 (+ 45us) client_negotiation.cc:769] Sending connection context
0620 11:42:32.053291 (+ 87us) client_negotiation.cc:240] Negotiation successful
0620 11:42:32.053389 (+ 98us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":124,"thread_start_us":78,"threads_started":1}
I20260620 11:42:32.053471 8248 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 11:42:32.047441 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.6.234.193:52603 (local address 127.6.234.254:43185)
0620 11:42:32.047479 (+ 38us) server_negotiation.cc:206] Beginning negotiation
0620 11:42:32.047481 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 11:42:32.047687 (+ 206us) server_negotiation.cc:407] Connection header received
0620 11:42:32.047751 (+ 64us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 11:42:32.047751 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 11:42:32.047760 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 11:42:32.047799 (+ 39us) server_negotiation.cc:226] Negotiated authn=SASL
0620 11:42:32.048144 (+ 345us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 11:42:32.048858 (+ 714us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 11:42:32.049944 (+ 1086us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 11:42:32.050216 (+ 272us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 11:42:32.050290 (+ 74us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 11:42:32.051145 (+ 855us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 11:42:32.051146 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 11:42:32.051146 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 11:42:32.051151 (+ 5us) server_negotiation.cc:922] calling sasl_server_start()
0620 11:42:32.052393 (+ 1242us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 11:42:32.052739 (+ 346us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 11:42:32.052740 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 11:42:32.052740 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 11:42:32.052793 (+ 53us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 11:42:32.052988 (+ 195us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 11:42:32.052989 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 11:42:32.052989 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 11:42:32.053091 (+ 102us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 11:42:32.053165 (+ 74us) server_negotiation.cc:1035] Waiting for connection context
0620 11:42:32.053314 (+ 149us) server_negotiation.cc:299] Negotiation successful
0620 11:42:32.053387 (+ 73us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":20}
WARNING: no policy specified for kudu/127.6.234.194@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.6.234.194@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.6.234.194 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task1JRg7N/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781955746769051-7083-0/krb5kdc/kudu_127.6.234.194.keytab.
Entry for principal kudu/127.6.234.194 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task1JRg7N/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781955746769051-7083-0/krb5kdc/kudu_127.6.234.194.keytab.
I20260620 11:42:32.072386 7083 external_mini_cluster.cc:1326] Running /tmp/dist-test-task1JRg7N/build/release/bin/kudu
/tmp/dist-test-task1JRg7N/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task1JRg7N/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781955746769051-7083-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-task1JRg7N/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781955746769051-7083-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task1JRg7N/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781955746769051-7083-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-task1JRg7N/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781955746769051-7083-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.6.234.194:0
--local_ip_for_outbound_sockets=127.6.234.194
--webserver_interface=127.6.234.194
--webserver_port=0
--tserver_master_addrs=127.6.234.254:43185
--keytab_file=/tmp/dist-test-task1JRg7N/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781955746769051-7083-0/krb5kdc/kudu_127.6.234.194.keytab
--principal=kudu/127.6.234.194
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.6.234.212:42185
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true with env {KRB5CCNAME=/tmp/dist-test-task1JRg7N/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781955746769051-7083-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task1JRg7N/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781955746769051-7083-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task1JRg7N/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781955746769051-7083-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 11:42:32.152215 8410 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 11:42:32.152436 8410 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 11:42:32.152474 8410 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 11:42:32.153687 8410 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 11:42:32.153765 8410 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 11:42:32.153801 8410 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260620 11:42:32.153841 8410 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.6.234.194
I20260620 11:42:32.155246 8410 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.6.234.212:42185
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task1JRg7N/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781955746769051-7083-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-task1JRg7N/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781955746769051-7083-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task1JRg7N/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781955746769051-7083-0/krb5kdc/kudu_127.6.234.194.keytab
--principal=kudu/127.6.234.194
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.6.234.194:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task1JRg7N/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781955746769051-7083-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.6.234.194
--webserver_port=0
--enable_txn_system_client_init=true
--tserver_master_addrs=127.6.234.254:43185
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.8410
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.6.234.194
--log_dir=/tmp/dist-test-task1JRg7N/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781955746769051-7083-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 20 Jun 2026 10:32:37 UTC on ccycloud.quasar-bgyofv.root.comops.site
build id 4457173
I20260620 11:42:32.155527 8410 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 11:42:32.155755 8410 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 11:42:32.160722 8415 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 11:42:32.160749 8416 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 11:42:32.160751 8418 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 11:42:32.161098 8410 server_base.cc:1034] running on GCE node
I20260620 11:42:32.161304 8410 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 11:42:32.161535 8410 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 11:42:32.162683 8410 hybrid_clock.cc:648] HybridClock initialized: now 1781955752162666 us; error 47 us; skew 500 ppm
Jun 20 11:42:32 dist-test-slave-cjc6 krb5kdc[8153](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781955752, etypes {rep=17 tkt=17 ses=17}, kudu/127.6.234.194@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 11:42:32.165287 8410 init.cc:377] Logged in from keytab as kudu/127.6.234.194@KRBTEST.COM (short username kudu)
I20260620 11:42:32.165827 8410 webserver.cc:460] Webserver started at http://127.6.234.194:42325/ using document root <none> and password file <none>
I20260620 11:42:32.166093 8410 fs_manager.cc:359] Metadata directory not provided
I20260620 11:42:32.166162 8410 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 11:42:32.166317 8410 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 11:42:32.167394 8410 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task1JRg7N/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781955746769051-7083-0/minicluster-data/ts-1/data/instance:
uuid: "08b45869e2ab438dba72a0323d9a56bb"
format_stamp: "Formatted at 2026-06-20 11:42:32 on dist-test-slave-cjc6"
server_key: "90092ae6c55ed38db7715c36e754dbf1"
server_key_iv: "c445cf1432a2e62a8bc2afe8e0603c32"
server_key_version: "encryptionkey@0"
I20260620 11:42:32.167747 8410 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task1JRg7N/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781955746769051-7083-0/minicluster-data/ts-1/wal/instance:
uuid: "08b45869e2ab438dba72a0323d9a56bb"
format_stamp: "Formatted at 2026-06-20 11:42:32 on dist-test-slave-cjc6"
server_key: "90092ae6c55ed38db7715c36e754dbf1"
server_key_iv: "c445cf1432a2e62a8bc2afe8e0603c32"
server_key_version: "encryptionkey@0"
I20260620 11:42:32.169102 8410 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.003s
I20260620 11:42:32.170008 8425 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 11:42:32.170171 8410 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.001s
I20260620 11:42:32.170270 8410 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task1JRg7N/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781955746769051-7083-0/minicluster-data/ts-1/data,/tmp/dist-test-task1JRg7N/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781955746769051-7083-0/minicluster-data/ts-1/wal
uuid: "08b45869e2ab438dba72a0323d9a56bb"
format_stamp: "Formatted at 2026-06-20 11:42:32 on dist-test-slave-cjc6"
server_key: "90092ae6c55ed38db7715c36e754dbf1"
server_key_iv: "c445cf1432a2e62a8bc2afe8e0603c32"
server_key_version: "encryptionkey@0"
I20260620 11:42:32.170351 8410 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task1JRg7N/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781955746769051-7083-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-task1JRg7N/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781955746769051-7083-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-task1JRg7N/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781955746769051-7083-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 11:42:32.185982 8410 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 11:42:32.188412 8410 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 11:42:32.188591 8410 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 11:42:32.189211 8410 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 11:42:32.189262 8410 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 11:42:32.189299 8410 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 11:42:32.189320 8410 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Jun 20 11:42:32 dist-test-slave-cjc6 krb5kdc[8153](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781955752, etypes {rep=17 tkt=17 ses=17}, kudu/127.6.234.194@KRBTEST.COM for kudu/127.6.234.254@KRBTEST.COM
I20260620 11:42:32.196656 8410 rpc_server.cc:307] RPC server started. Bound to: 127.6.234.194:37745
I20260620 11:42:32.197043 8410 server_base.cc:1166] Dumped server information to /tmp/dist-test-task1JRg7N/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781955746769051-7083-0/minicluster-data/ts-1/data/info.pb
I20260620 11:42:32.197077 8540 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.6.234.194:37745 every 8 connection(s)
I20260620 11:42:32.199044 8248 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 11:42:32.189776 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.6.234.194:59827 (local address 127.6.234.254:43185)
0620 11:42:32.189854 (+ 78us) server_negotiation.cc:206] Beginning negotiation
0620 11:42:32.189856 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 11:42:32.190083 (+ 227us) server_negotiation.cc:407] Connection header received
0620 11:42:32.190562 (+ 479us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 11:42:32.190564 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 11:42:32.190579 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 11:42:32.190670 (+ 91us) server_negotiation.cc:226] Negotiated authn=SASL
0620 11:42:32.191304 (+ 634us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 11:42:32.191802 (+ 498us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 11:42:32.192887 (+ 1085us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 11:42:32.193089 (+ 202us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 11:42:32.193162 (+ 73us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 11:42:32.196251 (+ 3089us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 11:42:32.196253 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 11:42:32.196253 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 11:42:32.196259 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0620 11:42:32.197791 (+ 1532us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 11:42:32.198184 (+ 393us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 11:42:32.198185 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 11:42:32.198185 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 11:42:32.198235 (+ 50us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 11:42:32.198435 (+ 200us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 11:42:32.198437 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 11:42:32.198437 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 11:42:32.198556 (+ 119us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 11:42:32.198622 (+ 66us) server_negotiation.cc:1035] Waiting for connection context
0620 11:42:32.198901 (+ 279us) server_negotiation.cc:299] Negotiation successful
0620 11:42:32.198980 (+ 79us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":42}
I20260620 11:42:32.199196 8434 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 11:42:32.189744 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.234.254:43185 (local address 127.6.234.194:59827)
0620 11:42:32.189946 (+ 202us) negotiation.cc:107] Waiting for socket to connect
0620 11:42:32.189987 (+ 41us) client_negotiation.cc:174] Beginning negotiation
0620 11:42:32.190469 (+ 482us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 11:42:32.190732 (+ 263us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 11:42:32.190736 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 11:42:32.191014 (+ 278us) client_negotiation.cc:189] Negotiated authn=SASL
0620 11:42:32.191203 (+ 189us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 11:42:32.191208 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 11:42:32.191911 (+ 703us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 11:42:32.191912 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 11:42:32.192787 (+ 875us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 11:42:32.192789 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 11:42:32.193181 (+ 392us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 11:42:32.193182 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 11:42:32.193335 (+ 153us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 11:42:32.193881 (+ 546us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 11:42:32.193890 (+ 9us) client_negotiation.cc:656] calling sasl_client_start()
0620 11:42:32.196159 (+ 2269us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 11:42:32.197881 (+ 1722us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 11:42:32.197885 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 11:42:32.197887 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0620 11:42:32.198120 (+ 233us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 11:42:32.198280 (+ 160us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 11:42:32.198281 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 11:42:32.198281 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0620 11:42:32.198362 (+ 81us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 11:42:32.198632 (+ 270us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 11:42:32.198643 (+ 11us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 11:42:32.198756 (+ 113us) client_negotiation.cc:769] Sending connection context
0620 11:42:32.198858 (+ 102us) client_negotiation.cc:240] Negotiation successful
0620 11:42:32.198999 (+ 141us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":160,"thread_start_us":127,"threads_started":1}
I20260620 11:42:32.199712 8541 heartbeater.cc:347] Connected to a master server at 127.6.234.254:43185
I20260620 11:42:32.199791 8541 heartbeater.cc:464] Registering TS with master...
I20260620 11:42:32.199977 8541 heartbeater.cc:511] Master 127.6.234.254:43185 requested a full tablet report, sending...
I20260620 11:42:32.200366 8197 ts_manager.cc:194] Registered new tserver with Master: 08b45869e2ab438dba72a0323d9a56bb (127.6.234.194:37745)
I20260620 11:42:32.200855 8197 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.6.234.194@KRBTEST.COM'} at 127.6.234.194:59827
I20260620 11:42:32.206051 8547 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 11:42:32.200773 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.234.254:43185 (local address 127.6.234.194:45921)
0620 11:42:32.200908 (+ 135us) negotiation.cc:107] Waiting for socket to connect
0620 11:42:32.200920 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0620 11:42:32.200990 (+ 70us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 11:42:32.201124 (+ 134us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 11:42:32.201125 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 11:42:32.201278 (+ 153us) client_negotiation.cc:189] Negotiated authn=SASL
0620 11:42:32.201325 (+ 47us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 11:42:32.201326 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 11:42:32.202202 (+ 876us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 11:42:32.202202 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 11:42:32.202707 (+ 505us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 11:42:32.202709 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 11:42:32.203007 (+ 298us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 11:42:32.203007 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 11:42:32.203061 (+ 54us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 11:42:32.203401 (+ 340us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 11:42:32.203404 (+ 3us) client_negotiation.cc:656] calling sasl_client_start()
0620 11:42:32.203656 (+ 252us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 11:42:32.205087 (+ 1431us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 11:42:32.205088 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 11:42:32.205088 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0620 11:42:32.205248 (+ 160us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 11:42:32.205418 (+ 170us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 11:42:32.205419 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 11:42:32.205419 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0620 11:42:32.205471 (+ 52us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 11:42:32.205782 (+ 311us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 11:42:32.205783 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 11:42:32.205834 (+ 51us) client_negotiation.cc:769] Sending connection context
0620 11:42:32.205904 (+ 70us) client_negotiation.cc:240] Negotiation successful
0620 11:42:32.205979 (+ 75us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":109,"thread_start_us":83,"threads_started":1}
I20260620 11:42:32.206102 8248 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 11:42:32.200866 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.6.234.194:45921 (local address 127.6.234.254:43185)
0620 11:42:32.200923 (+ 57us) server_negotiation.cc:206] Beginning negotiation
0620 11:42:32.200924 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0620 11:42:32.200976 (+ 52us) server_negotiation.cc:407] Connection header received
0620 11:42:32.201049 (+ 73us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 11:42:32.201050 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 11:42:32.201060 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 11:42:32.201122 (+ 62us) server_negotiation.cc:226] Negotiated authn=SASL
0620 11:42:32.201395 (+ 273us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 11:42:32.202116 (+ 721us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 11:42:32.202774 (+ 658us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 11:42:32.202944 (+ 170us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 11:42:32.202993 (+ 49us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 11:42:32.203748 (+ 755us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 11:42:32.203749 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 11:42:32.203749 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 11:42:32.203755 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0620 11:42:32.205003 (+ 1248us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 11:42:32.205315 (+ 312us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 11:42:32.205316 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 11:42:32.205316 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 11:42:32.205362 (+ 46us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 11:42:32.205524 (+ 162us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 11:42:32.205525 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 11:42:32.205525 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 11:42:32.205647 (+ 122us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 11:42:32.205753 (+ 106us) server_negotiation.cc:1035] Waiting for connection context
0620 11:42:32.205961 (+ 208us) server_negotiation.cc:299] Negotiation successful
0620 11:42:32.206037 (+ 76us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":20}
I20260620 11:42:32.206997 7083 external_mini_cluster.cc:1388] Started /tmp/dist-test-task1JRg7N/build/release/bin/kudu as pid 8410
I20260620 11:42:32.207083 7083 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task1JRg7N/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781955746769051-7083-0/minicluster-data/ts-1/wal/instance
I20260620 11:42:32.207284 7083 external_mini_cluster.cc:1428] Setting key ba2300ccef74f9a79d5b761ccd7ef1db
WARNING: no policy specified for kudu/127.6.234.195@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.6.234.195@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.6.234.195 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task1JRg7N/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781955746769051-7083-0/krb5kdc/kudu_127.6.234.195.keytab.
Entry for principal kudu/127.6.234.195 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task1JRg7N/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781955746769051-7083-0/krb5kdc/kudu_127.6.234.195.keytab.
I20260620 11:42:32.232478 7083 external_mini_cluster.cc:1326] Running /tmp/dist-test-task1JRg7N/build/release/bin/kudu
/tmp/dist-test-task1JRg7N/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task1JRg7N/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781955746769051-7083-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-task1JRg7N/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781955746769051-7083-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task1JRg7N/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781955746769051-7083-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-task1JRg7N/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781955746769051-7083-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.6.234.195:0
--local_ip_for_outbound_sockets=127.6.234.195
--webserver_interface=127.6.234.195
--webserver_port=0
--tserver_master_addrs=127.6.234.254:43185
--keytab_file=/tmp/dist-test-task1JRg7N/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781955746769051-7083-0/krb5kdc/kudu_127.6.234.195.keytab
--principal=kudu/127.6.234.195
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.6.234.212:42185
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true with env {KRB5CCNAME=/tmp/dist-test-task1JRg7N/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781955746769051-7083-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task1JRg7N/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781955746769051-7083-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task1JRg7N/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781955746769051-7083-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 11:42:32.314112 8550 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 11:42:32.314316 8550 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 11:42:32.314339 8550 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 11:42:32.315536 8550 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 11:42:32.315582 8550 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 11:42:32.315603 8550 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260620 11:42:32.315663 8550 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.6.234.195
I20260620 11:42:32.317070 8550 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.6.234.212:42185
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task1JRg7N/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781955746769051-7083-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-task1JRg7N/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781955746769051-7083-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task1JRg7N/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781955746769051-7083-0/krb5kdc/kudu_127.6.234.195.keytab
--principal=kudu/127.6.234.195
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.6.234.195:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task1JRg7N/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781955746769051-7083-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.6.234.195
--webserver_port=0
--enable_txn_system_client_init=true
--tserver_master_addrs=127.6.234.254:43185
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.8550
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.6.234.195
--log_dir=/tmp/dist-test-task1JRg7N/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781955746769051-7083-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 20 Jun 2026 10:32:37 UTC on ccycloud.quasar-bgyofv.root.comops.site
build id 4457173
I20260620 11:42:32.317313 8550 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 11:42:32.317555 8550 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 11:42:32.322471 8555 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 11:42:32.322429 8556 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 11:42:32.322569 8550 server_base.cc:1034] running on GCE node
W20260620 11:42:32.322428 8558 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 11:42:32.322850 8550 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 11:42:32.323079 8550 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 11:42:32.324216 8550 hybrid_clock.cc:648] HybridClock initialized: now 1781955752324200 us; error 27 us; skew 500 ppm
Jun 20 11:42:32 dist-test-slave-cjc6 krb5kdc[8153](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781955752, etypes {rep=17 tkt=17 ses=17}, kudu/127.6.234.195@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 11:42:32.326735 8550 init.cc:377] Logged in from keytab as kudu/127.6.234.195@KRBTEST.COM (short username kudu)
I20260620 11:42:32.327219 8550 webserver.cc:460] Webserver started at http://127.6.234.195:42911/ using document root <none> and password file <none>
I20260620 11:42:32.327437 8550 fs_manager.cc:359] Metadata directory not provided
I20260620 11:42:32.327504 8550 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 11:42:32.327648 8550 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 11:42:32.328677 8550 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task1JRg7N/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781955746769051-7083-0/minicluster-data/ts-2/data/instance:
uuid: "eab50d9c8b4e4cf6afe28767b40bd8f4"
format_stamp: "Formatted at 2026-06-20 11:42:32 on dist-test-slave-cjc6"
server_key: "0aa68cbe3ffadc7317082a50eead5e68"
server_key_iv: "37161fdc35488c5df622b19d5672d704"
server_key_version: "encryptionkey@0"
I20260620 11:42:32.329033 8550 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task1JRg7N/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781955746769051-7083-0/minicluster-data/ts-2/wal/instance:
uuid: "eab50d9c8b4e4cf6afe28767b40bd8f4"
format_stamp: "Formatted at 2026-06-20 11:42:32 on dist-test-slave-cjc6"
server_key: "0aa68cbe3ffadc7317082a50eead5e68"
server_key_iv: "37161fdc35488c5df622b19d5672d704"
server_key_version: "encryptionkey@0"
I20260620 11:42:32.330397 8550 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.003s
I20260620 11:42:32.331207 8565 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 11:42:32.331380 8550 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.001s
I20260620 11:42:32.331491 8550 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task1JRg7N/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781955746769051-7083-0/minicluster-data/ts-2/data,/tmp/dist-test-task1JRg7N/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781955746769051-7083-0/minicluster-data/ts-2/wal
uuid: "eab50d9c8b4e4cf6afe28767b40bd8f4"
format_stamp: "Formatted at 2026-06-20 11:42:32 on dist-test-slave-cjc6"
server_key: "0aa68cbe3ffadc7317082a50eead5e68"
server_key_iv: "37161fdc35488c5df622b19d5672d704"
server_key_version: "encryptionkey@0"
I20260620 11:42:32.331573 8550 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task1JRg7N/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781955746769051-7083-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-task1JRg7N/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781955746769051-7083-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-task1JRg7N/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781955746769051-7083-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 11:42:32.375916 8550 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 11:42:32.378443 8550 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 11:42:32.378610 8550 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 11:42:32.379208 8550 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 11:42:32.379274 8550 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 11:42:32.379323 8550 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 11:42:32.379374 8550 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Jun 20 11:42:32 dist-test-slave-cjc6 krb5kdc[8153](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781955752, etypes {rep=17 tkt=17 ses=17}, kudu/127.6.234.195@KRBTEST.COM for kudu/127.6.234.254@KRBTEST.COM
I20260620 11:42:32.386432 8550 rpc_server.cc:307] RPC server started. Bound to: 127.6.234.195:33411
I20260620 11:42:32.386530 8680 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.6.234.195:33411 every 8 connection(s)
I20260620 11:42:32.386863 8550 server_base.cc:1166] Dumped server information to /tmp/dist-test-task1JRg7N/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781955746769051-7083-0/minicluster-data/ts-2/data/info.pb
I20260620 11:42:32.387498 7083 external_mini_cluster.cc:1388] Started /tmp/dist-test-task1JRg7N/build/release/bin/kudu as pid 8550
I20260620 11:42:32.387588 7083 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task1JRg7N/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781955746769051-7083-0/minicluster-data/ts-2/wal/instance
I20260620 11:42:32.387764 7083 external_mini_cluster.cc:1428] Setting key 208ca69415d0f6593d22007ac4877442
I20260620 11:42:32.389463 8248 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 11:42:32.379824 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.6.234.195:37421 (local address 127.6.234.254:43185)
0620 11:42:32.379897 (+ 73us) server_negotiation.cc:206] Beginning negotiation
0620 11:42:32.379900 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0620 11:42:32.380150 (+ 250us) server_negotiation.cc:407] Connection header received
0620 11:42:32.380664 (+ 514us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 11:42:32.380666 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 11:42:32.380679 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 11:42:32.380768 (+ 89us) server_negotiation.cc:226] Negotiated authn=SASL
0620 11:42:32.381659 (+ 891us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 11:42:32.382557 (+ 898us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 11:42:32.383650 (+ 1093us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 11:42:32.383987 (+ 337us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 11:42:32.384076 (+ 89us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 11:42:32.386319 (+ 2243us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 11:42:32.386321 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 11:42:32.386321 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 11:42:32.386330 (+ 9us) server_negotiation.cc:922] calling sasl_server_start()
0620 11:42:32.388022 (+ 1692us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 11:42:32.388499 (+ 477us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 11:42:32.388500 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 11:42:32.388500 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 11:42:32.388558 (+ 58us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 11:42:32.388824 (+ 266us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 11:42:32.388825 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 11:42:32.388825 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 11:42:32.388947 (+ 122us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 11:42:32.389021 (+ 74us) server_negotiation.cc:1035] Waiting for connection context
0620 11:42:32.389285 (+ 264us) server_negotiation.cc:299] Negotiation successful
0620 11:42:32.389361 (+ 76us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":43}
I20260620 11:42:32.389645 8574 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 11:42:32.379794 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.234.254:43185 (local address 127.6.234.195:37421)
0620 11:42:32.380005 (+ 211us) negotiation.cc:107] Waiting for socket to connect
0620 11:42:32.380047 (+ 42us) client_negotiation.cc:174] Beginning negotiation
0620 11:42:32.380519 (+ 472us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 11:42:32.380883 (+ 364us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 11:42:32.380888 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 11:42:32.381247 (+ 359us) client_negotiation.cc:189] Negotiated authn=SASL
0620 11:42:32.381544 (+ 297us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 11:42:32.381550 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 11:42:32.382659 (+ 1109us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 11:42:32.382660 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 11:42:32.383580 (+ 920us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 11:42:32.383583 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 11:42:32.384092 (+ 509us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 11:42:32.384093 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 11:42:32.384230 (+ 137us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 11:42:32.384764 (+ 534us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 11:42:32.384772 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0620 11:42:32.386198 (+ 1426us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 11:42:32.388121 (+ 1923us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 11:42:32.388128 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 11:42:32.388130 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0620 11:42:32.388410 (+ 280us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 11:42:32.388633 (+ 223us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 11:42:32.388634 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 11:42:32.388634 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0620 11:42:32.388747 (+ 113us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 11:42:32.389012 (+ 265us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 11:42:32.389018 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 11:42:32.389157 (+ 139us) client_negotiation.cc:769] Sending connection context
0620 11:42:32.389266 (+ 109us) client_negotiation.cc:240] Negotiation successful
0620 11:42:32.389421 (+ 155us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":173,"thread_start_us":94,"threads_started":1}
I20260620 11:42:32.390105 8681 heartbeater.cc:347] Connected to a master server at 127.6.234.254:43185
I20260620 11:42:32.390199 8681 heartbeater.cc:464] Registering TS with master...
I20260620 11:42:32.390368 8681 heartbeater.cc:511] Master 127.6.234.254:43185 requested a full tablet report, sending...
I20260620 11:42:32.390717 8197 ts_manager.cc:194] Registered new tserver with Master: eab50d9c8b4e4cf6afe28767b40bd8f4 (127.6.234.195:33411)
I20260620 11:42:32.391180 8197 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.6.234.195@KRBTEST.COM'} at 127.6.234.195:37421
I20260620 11:42:32.396282 8248 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 11:42:32.391177 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.6.234.195:40393 (local address 127.6.234.254:43185)
0620 11:42:32.391220 (+ 43us) server_negotiation.cc:206] Beginning negotiation
0620 11:42:32.391222 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 11:42:32.391325 (+ 103us) server_negotiation.cc:407] Connection header received
0620 11:42:32.391414 (+ 89us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 11:42:32.391415 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 11:42:32.391426 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 11:42:32.391486 (+ 60us) server_negotiation.cc:226] Negotiated authn=SASL
0620 11:42:32.391823 (+ 337us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 11:42:32.392324 (+ 501us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 11:42:32.392967 (+ 643us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 11:42:32.393154 (+ 187us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 11:42:32.393215 (+ 61us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 11:42:32.394042 (+ 827us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 11:42:32.394043 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 11:42:32.394043 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 11:42:32.394049 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0620 11:42:32.395485 (+ 1436us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 11:42:32.395760 (+ 275us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 11:42:32.395760 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 11:42:32.395760 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 11:42:32.395800 (+ 40us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 11:42:32.395950 (+ 150us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 11:42:32.395950 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 11:42:32.395950 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 11:42:32.396030 (+ 80us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 11:42:32.396073 (+ 43us) server_negotiation.cc:1035] Waiting for connection context
0620 11:42:32.396177 (+ 104us) server_negotiation.cc:299] Negotiation successful
0620 11:42:32.396226 (+ 49us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":21}
I20260620 11:42:32.396294 8687 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 11:42:32.391141 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.234.254:43185 (local address 127.6.234.195:40393)
0620 11:42:32.391265 (+ 124us) negotiation.cc:107] Waiting for socket to connect
0620 11:42:32.391277 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0620 11:42:32.391336 (+ 59us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 11:42:32.391504 (+ 168us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 11:42:32.391505 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 11:42:32.391687 (+ 182us) client_negotiation.cc:189] Negotiated authn=SASL
0620 11:42:32.391744 (+ 57us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 11:42:32.391745 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 11:42:32.392405 (+ 660us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 11:42:32.392406 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 11:42:32.392886 (+ 480us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 11:42:32.392887 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 11:42:32.393224 (+ 337us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 11:42:32.393225 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 11:42:32.393278 (+ 53us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 11:42:32.393610 (+ 332us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 11:42:32.393614 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0620 11:42:32.393951 (+ 337us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 11:42:32.395556 (+ 1605us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 11:42:32.395556 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 11:42:32.395557 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0620 11:42:32.395701 (+ 144us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 11:42:32.395851 (+ 150us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 11:42:32.395851 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 11:42:32.395851 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0620 11:42:32.395897 (+ 46us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 11:42:32.396077 (+ 180us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 11:42:32.396078 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 11:42:32.396111 (+ 33us) client_negotiation.cc:769] Sending connection context
0620 11:42:32.396161 (+ 50us) client_negotiation.cc:240] Negotiation successful
0620 11:42:32.396229 (+ 68us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":97,"thread_start_us":67,"threads_started":1}
I20260620 11:42:32.399354 7083 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
Jun 20 11:42:32 dist-test-slave-cjc6 krb5kdc[8153](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781955752, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-user@KRBTEST.COM:
Jun 20 11:42:32 dist-test-slave-cjc6 krb5kdc[8153](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781955752, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for kudu/127.6.234.254@KRBTEST.COM
I20260620 11:42:32.420820 8248 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 11:42:32.414350 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46864 (local address 127.6.234.254:43185)
0620 11:42:32.414412 (+ 62us) server_negotiation.cc:206] Beginning negotiation
0620 11:42:32.414414 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 11:42:32.414424 (+ 10us) server_negotiation.cc:407] Connection header received
0620 11:42:32.414462 (+ 38us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 11:42:32.414462 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 11:42:32.414472 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 11:42:32.414542 (+ 70us) server_negotiation.cc:226] Negotiated authn=SASL
0620 11:42:32.415080 (+ 538us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 11:42:32.415579 (+ 499us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 11:42:32.416331 (+ 752us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 11:42:32.416527 (+ 196us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 11:42:32.416602 (+ 75us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 11:42:32.418797 (+ 2195us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 11:42:32.418798 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 11:42:32.418799 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 11:42:32.418804 (+ 5us) server_negotiation.cc:922] calling sasl_server_start()
0620 11:42:32.419952 (+ 1148us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 11:42:32.420255 (+ 303us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 11:42:32.420256 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 11:42:32.420256 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 11:42:32.420301 (+ 45us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 11:42:32.420460 (+ 159us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 11:42:32.420461 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 11:42:32.420461 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 11:42:32.420553 (+ 92us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 11:42:32.420627 (+ 74us) server_negotiation.cc:1035] Waiting for connection context
0620 11:42:32.420711 (+ 84us) server_negotiation.cc:299] Negotiation successful
0620 11:42:32.420761 (+ 50us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":36}
I20260620 11:42:32.422401 8197 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:46864:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260620 11:42:32.423995 8197 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260620 11:42:32.431150 8698 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 11:42:32.426534 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.234.194:37745 (local address 127.0.0.1:52026)
0620 11:42:32.426832 (+ 298us) negotiation.cc:107] Waiting for socket to connect
0620 11:42:32.427057 (+ 225us) client_negotiation.cc:174] Beginning negotiation
0620 11:42:32.427140 (+ 83us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 11:42:32.427439 (+ 299us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 11:42:32.427440 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 11:42:32.427441 (+ 1us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 11:42:32.427481 (+ 40us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 11:42:32.427483 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 11:42:32.428793 (+ 1310us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 11:42:32.428794 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 11:42:32.430162 (+ 1368us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 11:42:32.430165 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 11:42:32.430849 (+ 684us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 11:42:32.430850 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 11:42:32.430926 (+ 76us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 11:42:32.430940 (+ 14us) client_negotiation.cc:769] Sending connection context
0620 11:42:32.431006 (+ 66us) client_negotiation.cc:240] Negotiation successful
0620 11:42:32.431054 (+ 48us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":271,"mutex_wait_us":48,"thread_start_us":60,"threads_started":1}
I20260620 11:42:32.431255 8701 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 11:42:32.426786 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.234.195:33411 (local address 127.0.0.1:49788)
0620 11:42:32.427575 (+ 789us) negotiation.cc:107] Waiting for socket to connect
0620 11:42:32.427583 (+ 8us) client_negotiation.cc:174] Beginning negotiation
0620 11:42:32.427642 (+ 59us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 11:42:32.427830 (+ 188us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 11:42:32.427830 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 11:42:32.427832 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 11:42:32.427870 (+ 38us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 11:42:32.427871 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 11:42:32.429086 (+ 1215us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 11:42:32.429087 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 11:42:32.430018 (+ 931us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 11:42:32.430021 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 11:42:32.431018 (+ 997us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 11:42:32.431019 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 11:42:32.431083 (+ 64us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 11:42:32.431088 (+ 5us) client_negotiation.cc:769] Sending connection context
0620 11:42:32.431137 (+ 49us) client_negotiation.cc:240] Negotiation successful
0620 11:42:32.431177 (+ 40us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":767,"thread_start_us":74,"threads_started":1}
I20260620 11:42:32.431412 8699 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 11:42:32.426534 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.234.193:42909 (local address 127.0.0.1:57570)
0620 11:42:32.427121 (+ 587us) negotiation.cc:107] Waiting for socket to connect
0620 11:42:32.427131 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0620 11:42:32.427183 (+ 52us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 11:42:32.427377 (+ 194us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 11:42:32.427380 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 11:42:32.427383 (+ 3us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 11:42:32.427444 (+ 61us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 11:42:32.427446 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 11:42:32.428794 (+ 1348us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 11:42:32.428795 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 11:42:32.430162 (+ 1367us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 11:42:32.430165 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 11:42:32.431007 (+ 842us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 11:42:32.431007 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 11:42:32.431070 (+ 63us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 11:42:32.431073 (+ 3us) client_negotiation.cc:769] Sending connection context
0620 11:42:32.431125 (+ 52us) client_negotiation.cc:240] Negotiation successful
0620 11:42:32.431155 (+ 30us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":565,"mutex_wait_us":172,"thread_start_us":84,"threads_started":1}
I20260620 11:42:32.431556 8703 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 11:42:32.426951 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57570 (local address 127.6.234.193:42909)
0620 11:42:32.427105 (+ 154us) server_negotiation.cc:206] Beginning negotiation
0620 11:42:32.427109 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 11:42:32.427237 (+ 128us) server_negotiation.cc:407] Connection header received
0620 11:42:32.427281 (+ 44us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 11:42:32.427284 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 11:42:32.427299 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 11:42:32.427377 (+ 78us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 11:42:32.427509 (+ 132us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 11:42:32.428707 (+ 1198us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 11:42:32.430244 (+ 1537us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 11:42:32.430945 (+ 701us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 11:42:32.431010 (+ 65us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 11:42:32.431056 (+ 46us) server_negotiation.cc:1035] Waiting for connection context
0620 11:42:32.431136 (+ 80us) server_negotiation.cc:299] Negotiation successful
0620 11:42:32.431172 (+ 36us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":124,"thread_start_us":79,"threads_started":1}
I20260620 11:42:32.431573 8700 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 11:42:32.426631 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52026 (local address 127.6.234.194:37745)
0620 11:42:32.426839 (+ 208us) server_negotiation.cc:206] Beginning negotiation
0620 11:42:32.426844 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 11:42:32.427119 (+ 275us) server_negotiation.cc:407] Connection header received
0620 11:42:32.427265 (+ 146us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 11:42:32.427268 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 11:42:32.427285 (+ 17us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 11:42:32.427338 (+ 53us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 11:42:32.427554 (+ 216us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 11:42:32.428688 (+ 1134us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 11:42:32.430261 (+ 1573us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 11:42:32.430745 (+ 484us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 11:42:32.430846 (+ 101us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 11:42:32.430910 (+ 64us) server_negotiation.cc:1035] Waiting for connection context
0620 11:42:32.431404 (+ 494us) server_negotiation.cc:299] Negotiation successful
0620 11:42:32.431459 (+ 55us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":171,"thread_start_us":100,"threads_started":1}
I20260620 11:42:32.431751 8702 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 11:42:32.426885 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49788 (local address 127.6.234.195:33411)
0620 11:42:32.427018 (+ 133us) server_negotiation.cc:206] Beginning negotiation
0620 11:42:32.427021 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0620 11:42:32.427644 (+ 623us) server_negotiation.cc:407] Connection header received
0620 11:42:32.427709 (+ 65us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 11:42:32.427711 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 11:42:32.427725 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 11:42:32.427875 (+ 150us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 11:42:32.427982 (+ 107us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 11:42:32.429003 (+ 1021us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 11:42:32.430107 (+ 1104us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 11:42:32.430837 (+ 730us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 11:42:32.430920 (+ 83us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 11:42:32.430979 (+ 59us) server_negotiation.cc:1035] Waiting for connection context
0620 11:42:32.431336 (+ 357us) server_negotiation.cc:299] Negotiation successful
0620 11:42:32.431666 (+ 330us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":109,"thread_start_us":82,"threads_started":1}
I20260620 11:42:32.432332 8615 tablet_service.cc:1467] Processing CreateTablet for tablet a64cfba0d8344082a7fd69dd5ac3c94f (DEFAULT_TABLE table=test-table [id=6fd0eb701fde4a8eb98bb6938552469d]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 11:42:32.432360 8475 tablet_service.cc:1467] Processing CreateTablet for tablet a64cfba0d8344082a7fd69dd5ac3c94f (DEFAULT_TABLE table=test-table [id=6fd0eb701fde4a8eb98bb6938552469d]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 11:42:32.432360 8335 tablet_service.cc:1467] Processing CreateTablet for tablet a64cfba0d8344082a7fd69dd5ac3c94f (DEFAULT_TABLE table=test-table [id=6fd0eb701fde4a8eb98bb6938552469d]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 11:42:32.432663 8615 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a64cfba0d8344082a7fd69dd5ac3c94f. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 11:42:32.432686 8475 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a64cfba0d8344082a7fd69dd5ac3c94f. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 11:42:32.432686 8335 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a64cfba0d8344082a7fd69dd5ac3c94f. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 11:42:32.435396 8704 tablet_bootstrap.cc:492] T a64cfba0d8344082a7fd69dd5ac3c94f P eab50d9c8b4e4cf6afe28767b40bd8f4: Bootstrap starting.
I20260620 11:42:32.436271 8704 tablet_bootstrap.cc:654] T a64cfba0d8344082a7fd69dd5ac3c94f P eab50d9c8b4e4cf6afe28767b40bd8f4: Neither blocks nor log segments found. Creating new log.
I20260620 11:42:32.436295 8706 tablet_bootstrap.cc:492] T a64cfba0d8344082a7fd69dd5ac3c94f P df86ed971c13424aba5fe002377e02e8: Bootstrap starting.
I20260620 11:42:32.436731 8704 log.cc:824] T a64cfba0d8344082a7fd69dd5ac3c94f P eab50d9c8b4e4cf6afe28767b40bd8f4: Log is configured to *not* fsync() on all Append() calls
I20260620 11:42:32.436908 8706 tablet_bootstrap.cc:654] T a64cfba0d8344082a7fd69dd5ac3c94f P df86ed971c13424aba5fe002377e02e8: Neither blocks nor log segments found. Creating new log.
I20260620 11:42:32.437247 8706 log.cc:824] T a64cfba0d8344082a7fd69dd5ac3c94f P df86ed971c13424aba5fe002377e02e8: Log is configured to *not* fsync() on all Append() calls
I20260620 11:42:32.437518 8704 tablet_bootstrap.cc:492] T a64cfba0d8344082a7fd69dd5ac3c94f P eab50d9c8b4e4cf6afe28767b40bd8f4: No bootstrap required, opened a new log
I20260620 11:42:32.437563 8705 tablet_bootstrap.cc:492] T a64cfba0d8344082a7fd69dd5ac3c94f P 08b45869e2ab438dba72a0323d9a56bb: Bootstrap starting.
I20260620 11:42:32.437588 8704 ts_tablet_manager.cc:1397] T a64cfba0d8344082a7fd69dd5ac3c94f P eab50d9c8b4e4cf6afe28767b40bd8f4: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20260620 11:42:32.437963 8706 tablet_bootstrap.cc:492] T a64cfba0d8344082a7fd69dd5ac3c94f P df86ed971c13424aba5fe002377e02e8: No bootstrap required, opened a new log
I20260620 11:42:32.438042 8706 ts_tablet_manager.cc:1397] T a64cfba0d8344082a7fd69dd5ac3c94f P df86ed971c13424aba5fe002377e02e8: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260620 11:42:32.438176 8705 tablet_bootstrap.cc:654] T a64cfba0d8344082a7fd69dd5ac3c94f P 08b45869e2ab438dba72a0323d9a56bb: Neither blocks nor log segments found. Creating new log.
I20260620 11:42:32.438495 8705 log.cc:824] T a64cfba0d8344082a7fd69dd5ac3c94f P 08b45869e2ab438dba72a0323d9a56bb: Log is configured to *not* fsync() on all Append() calls
I20260620 11:42:32.439285 8705 tablet_bootstrap.cc:492] T a64cfba0d8344082a7fd69dd5ac3c94f P 08b45869e2ab438dba72a0323d9a56bb: No bootstrap required, opened a new log
I20260620 11:42:32.439354 8705 ts_tablet_manager.cc:1397] T a64cfba0d8344082a7fd69dd5ac3c94f P 08b45869e2ab438dba72a0323d9a56bb: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260620 11:42:32.439743 8704 raft_consensus.cc:348] T a64cfba0d8344082a7fd69dd5ac3c94f P eab50d9c8b4e4cf6afe28767b40bd8f4 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "08b45869e2ab438dba72a0323d9a56bb" member_type: VOTER last_known_addr { host: "127.6.234.194" port: 37745 } } peers { permanent_uuid: "eab50d9c8b4e4cf6afe28767b40bd8f4" member_type: VOTER last_known_addr { host: "127.6.234.195" port: 33411 } } peers { permanent_uuid: "df86ed971c13424aba5fe002377e02e8" member_type: VOTER last_known_addr { host: "127.6.234.193" port: 42909 } }
I20260620 11:42:32.439906 8704 raft_consensus.cc:374] T a64cfba0d8344082a7fd69dd5ac3c94f P eab50d9c8b4e4cf6afe28767b40bd8f4 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 11:42:32.439939 8704 raft_consensus.cc:729] T a64cfba0d8344082a7fd69dd5ac3c94f P eab50d9c8b4e4cf6afe28767b40bd8f4 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: eab50d9c8b4e4cf6afe28767b40bd8f4, State: Initialized, Role: FOLLOWER
I20260620 11:42:32.439813 8706 raft_consensus.cc:348] T a64cfba0d8344082a7fd69dd5ac3c94f P df86ed971c13424aba5fe002377e02e8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "08b45869e2ab438dba72a0323d9a56bb" member_type: VOTER last_known_addr { host: "127.6.234.194" port: 37745 } } peers { permanent_uuid: "eab50d9c8b4e4cf6afe28767b40bd8f4" member_type: VOTER last_known_addr { host: "127.6.234.195" port: 33411 } } peers { permanent_uuid: "df86ed971c13424aba5fe002377e02e8" member_type: VOTER last_known_addr { host: "127.6.234.193" port: 42909 } }
I20260620 11:42:32.439980 8706 raft_consensus.cc:374] T a64cfba0d8344082a7fd69dd5ac3c94f P df86ed971c13424aba5fe002377e02e8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 11:42:32.440016 8706 raft_consensus.cc:729] T a64cfba0d8344082a7fd69dd5ac3c94f P df86ed971c13424aba5fe002377e02e8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: df86ed971c13424aba5fe002377e02e8, State: Initialized, Role: FOLLOWER
I20260620 11:42:32.440073 8704 consensus_queue.cc:260] T a64cfba0d8344082a7fd69dd5ac3c94f P eab50d9c8b4e4cf6afe28767b40bd8f4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "08b45869e2ab438dba72a0323d9a56bb" member_type: VOTER last_known_addr { host: "127.6.234.194" port: 37745 } } peers { permanent_uuid: "eab50d9c8b4e4cf6afe28767b40bd8f4" member_type: VOTER last_known_addr { host: "127.6.234.195" port: 33411 } } peers { permanent_uuid: "df86ed971c13424aba5fe002377e02e8" member_type: VOTER last_known_addr { host: "127.6.234.193" port: 42909 } }
I20260620 11:42:32.440140 8706 consensus_queue.cc:260] T a64cfba0d8344082a7fd69dd5ac3c94f P df86ed971c13424aba5fe002377e02e8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "08b45869e2ab438dba72a0323d9a56bb" member_type: VOTER last_known_addr { host: "127.6.234.194" port: 37745 } } peers { permanent_uuid: "eab50d9c8b4e4cf6afe28767b40bd8f4" member_type: VOTER last_known_addr { host: "127.6.234.195" port: 33411 } } peers { permanent_uuid: "df86ed971c13424aba5fe002377e02e8" member_type: VOTER last_known_addr { host: "127.6.234.193" port: 42909 } }
I20260620 11:42:32.440318 8704 ts_tablet_manager.cc:1428] T a64cfba0d8344082a7fd69dd5ac3c94f P eab50d9c8b4e4cf6afe28767b40bd8f4: Time spent starting tablet: real 0.003s user 0.000s sys 0.003s
I20260620 11:42:32.440449 8681 heartbeater.cc:503] Master 127.6.234.254:43185 was elected leader, sending a full tablet report...
I20260620 11:42:32.440680 8401 heartbeater.cc:503] Master 127.6.234.254:43185 was elected leader, sending a full tablet report...
I20260620 11:42:32.441047 8706 ts_tablet_manager.cc:1428] T a64cfba0d8344082a7fd69dd5ac3c94f P df86ed971c13424aba5fe002377e02e8: Time spent starting tablet: real 0.003s user 0.000s sys 0.003s
I20260620 11:42:32.441325 8705 raft_consensus.cc:348] T a64cfba0d8344082a7fd69dd5ac3c94f P 08b45869e2ab438dba72a0323d9a56bb [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "08b45869e2ab438dba72a0323d9a56bb" member_type: VOTER last_known_addr { host: "127.6.234.194" port: 37745 } } peers { permanent_uuid: "eab50d9c8b4e4cf6afe28767b40bd8f4" member_type: VOTER last_known_addr { host: "127.6.234.195" port: 33411 } } peers { permanent_uuid: "df86ed971c13424aba5fe002377e02e8" member_type: VOTER last_known_addr { host: "127.6.234.193" port: 42909 } }
I20260620 11:42:32.441471 8705 raft_consensus.cc:374] T a64cfba0d8344082a7fd69dd5ac3c94f P 08b45869e2ab438dba72a0323d9a56bb [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 11:42:32.441512 8705 raft_consensus.cc:729] T a64cfba0d8344082a7fd69dd5ac3c94f P 08b45869e2ab438dba72a0323d9a56bb [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 08b45869e2ab438dba72a0323d9a56bb, State: Initialized, Role: FOLLOWER
I20260620 11:42:32.441630 8705 consensus_queue.cc:260] T a64cfba0d8344082a7fd69dd5ac3c94f P 08b45869e2ab438dba72a0323d9a56bb [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "08b45869e2ab438dba72a0323d9a56bb" member_type: VOTER last_known_addr { host: "127.6.234.194" port: 37745 } } peers { permanent_uuid: "eab50d9c8b4e4cf6afe28767b40bd8f4" member_type: VOTER last_known_addr { host: "127.6.234.195" port: 33411 } } peers { permanent_uuid: "df86ed971c13424aba5fe002377e02e8" member_type: VOTER last_known_addr { host: "127.6.234.193" port: 42909 } }
I20260620 11:42:32.441901 8705 ts_tablet_manager.cc:1428] T a64cfba0d8344082a7fd69dd5ac3c94f P 08b45869e2ab438dba72a0323d9a56bb: Time spent starting tablet: real 0.003s user 0.003s sys 0.000s
I20260620 11:42:32.441963 8541 heartbeater.cc:503] Master 127.6.234.254:43185 was elected leader, sending a full tablet report...
W20260620 11:42:32.447558 8542 tablet.cc:2369] T a64cfba0d8344082a7fd69dd5ac3c94f P 08b45869e2ab438dba72a0323d9a56bb: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260620 11:42:32.457953 8711 raft_consensus.cc:482] T a64cfba0d8344082a7fd69dd5ac3c94f P df86ed971c13424aba5fe002377e02e8 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260620 11:42:32.458094 8711 raft_consensus.cc:504] T a64cfba0d8344082a7fd69dd5ac3c94f P df86ed971c13424aba5fe002377e02e8 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "08b45869e2ab438dba72a0323d9a56bb" member_type: VOTER last_known_addr { host: "127.6.234.194" port: 37745 } } peers { permanent_uuid: "eab50d9c8b4e4cf6afe28767b40bd8f4" member_type: VOTER last_known_addr { host: "127.6.234.195" port: 33411 } } peers { permanent_uuid: "df86ed971c13424aba5fe002377e02e8" member_type: VOTER last_known_addr { host: "127.6.234.193" port: 42909 } }
I20260620 11:42:32.458463 8711 leader_election.cc:288] T a64cfba0d8344082a7fd69dd5ac3c94f P df86ed971c13424aba5fe002377e02e8 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 08b45869e2ab438dba72a0323d9a56bb (127.6.234.194:37745), eab50d9c8b4e4cf6afe28767b40bd8f4 (127.6.234.195:33411)
I20260620 11:42:32.461284 8294 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 11:42:32.458673 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.234.194:37745 (local address 127.6.234.193:45307)
0620 11:42:32.458735 (+ 62us) negotiation.cc:107] Waiting for socket to connect
0620 11:42:32.458749 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0620 11:42:32.458806 (+ 57us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 11:42:32.458998 (+ 192us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 11:42:32.458999 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 11:42:32.459001 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 11:42:32.459075 (+ 74us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 11:42:32.459076 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 11:42:32.459696 (+ 620us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 11:42:32.459697 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 11:42:32.460447 (+ 750us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 11:42:32.460449 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 11:42:32.461017 (+ 568us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 11:42:32.461018 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 11:42:32.461106 (+ 88us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 11:42:32.461109 (+ 3us) client_negotiation.cc:769] Sending connection context
0620 11:42:32.461174 (+ 65us) client_negotiation.cc:240] Negotiation successful
0620 11:42:32.461216 (+ 42us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":40}
I20260620 11:42:32.461283 8700 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 11:42:32.458762 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.6.234.193:45307 (local address 127.6.234.194:37745)
0620 11:42:32.458844 (+ 82us) server_negotiation.cc:206] Beginning negotiation
0620 11:42:32.458846 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 11:42:32.458866 (+ 20us) server_negotiation.cc:407] Connection header received
0620 11:42:32.458896 (+ 30us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 11:42:32.458897 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 11:42:32.458907 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 11:42:32.458994 (+ 87us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 11:42:32.459150 (+ 156us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 11:42:32.459616 (+ 466us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 11:42:32.460541 (+ 925us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 11:42:32.460930 (+ 389us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 11:42:32.461011 (+ 81us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 11:42:32.461020 (+ 9us) server_negotiation.cc:1035] Waiting for connection context
0620 11:42:32.461181 (+ 161us) server_negotiation.cc:299] Negotiation successful
0620 11:42:32.461216 (+ 35us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":47}
I20260620 11:42:32.461450 8702 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 11:42:32.458938 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.6.234.193:54795 (local address 127.6.234.195:33411)
0620 11:42:32.458986 (+ 48us) server_negotiation.cc:206] Beginning negotiation
0620 11:42:32.458988 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 11:42:32.459100 (+ 112us) server_negotiation.cc:407] Connection header received
0620 11:42:32.459159 (+ 59us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 11:42:32.459160 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 11:42:32.459167 (+ 7us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 11:42:32.459234 (+ 67us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 11:42:32.459392 (+ 158us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 11:42:32.459850 (+ 458us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 11:42:32.460742 (+ 892us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 11:42:32.461107 (+ 365us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 11:42:32.461191 (+ 84us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 11:42:32.461198 (+ 7us) server_negotiation.cc:1035] Waiting for connection context
0620 11:42:32.461349 (+ 151us) server_negotiation.cc:299] Negotiation successful
0620 11:42:32.461384 (+ 35us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":27}
I20260620 11:42:32.461473 8713 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 11:42:32.458847 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.234.195:33411 (local address 127.6.234.193:54795)
0620 11:42:32.459031 (+ 184us) negotiation.cc:107] Waiting for socket to connect
0620 11:42:32.459040 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0620 11:42:32.459096 (+ 56us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 11:42:32.459238 (+ 142us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 11:42:32.459239 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 11:42:32.459240 (+ 1us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 11:42:32.459311 (+ 71us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 11:42:32.459313 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 11:42:32.459932 (+ 619us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 11:42:32.459933 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 11:42:32.460650 (+ 717us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 11:42:32.460651 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 11:42:32.461189 (+ 538us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 11:42:32.461190 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 11:42:32.461265 (+ 75us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 11:42:32.461268 (+ 3us) client_negotiation.cc:769] Sending connection context
0620 11:42:32.461344 (+ 76us) client_negotiation.cc:240] Negotiation successful
0620 11:42:32.461384 (+ 40us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":170,"thread_start_us":112,"threads_started":1}
I20260620 11:42:32.461638 8495 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a64cfba0d8344082a7fd69dd5ac3c94f" candidate_uuid: "df86ed971c13424aba5fe002377e02e8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "08b45869e2ab438dba72a0323d9a56bb" is_pre_election: true
I20260620 11:42:32.461830 8495 raft_consensus.cc:2454] T a64cfba0d8344082a7fd69dd5ac3c94f P 08b45869e2ab438dba72a0323d9a56bb [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate df86ed971c13424aba5fe002377e02e8 in term 0.
I20260620 11:42:32.462116 8289 leader_election.cc:302] T a64cfba0d8344082a7fd69dd5ac3c94f P df86ed971c13424aba5fe002377e02e8 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 08b45869e2ab438dba72a0323d9a56bb, df86ed971c13424aba5fe002377e02e8; no voters:
I20260620 11:42:32.462203 8635 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a64cfba0d8344082a7fd69dd5ac3c94f" candidate_uuid: "df86ed971c13424aba5fe002377e02e8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "eab50d9c8b4e4cf6afe28767b40bd8f4" is_pre_election: true
I20260620 11:42:32.462296 8711 raft_consensus.cc:2781] T a64cfba0d8344082a7fd69dd5ac3c94f P df86ed971c13424aba5fe002377e02e8 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260620 11:42:32.462357 8635 raft_consensus.cc:2454] T a64cfba0d8344082a7fd69dd5ac3c94f P eab50d9c8b4e4cf6afe28767b40bd8f4 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate df86ed971c13424aba5fe002377e02e8 in term 0.
I20260620 11:42:32.462371 8711 raft_consensus.cc:482] T a64cfba0d8344082a7fd69dd5ac3c94f P df86ed971c13424aba5fe002377e02e8 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260620 11:42:32.462402 8711 raft_consensus.cc:3037] T a64cfba0d8344082a7fd69dd5ac3c94f P df86ed971c13424aba5fe002377e02e8 [term 0 FOLLOWER]: Advancing to term 1
I20260620 11:42:32.463135 8711 raft_consensus.cc:504] T a64cfba0d8344082a7fd69dd5ac3c94f P df86ed971c13424aba5fe002377e02e8 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "08b45869e2ab438dba72a0323d9a56bb" member_type: VOTER last_known_addr { host: "127.6.234.194" port: 37745 } } peers { permanent_uuid: "eab50d9c8b4e4cf6afe28767b40bd8f4" member_type: VOTER last_known_addr { host: "127.6.234.195" port: 33411 } } peers { permanent_uuid: "df86ed971c13424aba5fe002377e02e8" member_type: VOTER last_known_addr { host: "127.6.234.193" port: 42909 } }
I20260620 11:42:32.463274 8711 leader_election.cc:288] T a64cfba0d8344082a7fd69dd5ac3c94f P df86ed971c13424aba5fe002377e02e8 [CANDIDATE]: Term 1 election: Requested vote from peers 08b45869e2ab438dba72a0323d9a56bb (127.6.234.194:37745), eab50d9c8b4e4cf6afe28767b40bd8f4 (127.6.234.195:33411)
I20260620 11:42:32.463450 8495 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a64cfba0d8344082a7fd69dd5ac3c94f" candidate_uuid: "df86ed971c13424aba5fe002377e02e8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "08b45869e2ab438dba72a0323d9a56bb"
I20260620 11:42:32.463480 8635 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a64cfba0d8344082a7fd69dd5ac3c94f" candidate_uuid: "df86ed971c13424aba5fe002377e02e8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "eab50d9c8b4e4cf6afe28767b40bd8f4"
I20260620 11:42:32.463534 8495 raft_consensus.cc:3037] T a64cfba0d8344082a7fd69dd5ac3c94f P 08b45869e2ab438dba72a0323d9a56bb [term 0 FOLLOWER]: Advancing to term 1
I20260620 11:42:32.463563 8635 raft_consensus.cc:3037] T a64cfba0d8344082a7fd69dd5ac3c94f P eab50d9c8b4e4cf6afe28767b40bd8f4 [term 0 FOLLOWER]: Advancing to term 1
I20260620 11:42:32.464311 8495 raft_consensus.cc:2454] T a64cfba0d8344082a7fd69dd5ac3c94f P 08b45869e2ab438dba72a0323d9a56bb [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate df86ed971c13424aba5fe002377e02e8 in term 1.
I20260620 11:42:32.464432 8635 raft_consensus.cc:2454] T a64cfba0d8344082a7fd69dd5ac3c94f P eab50d9c8b4e4cf6afe28767b40bd8f4 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate df86ed971c13424aba5fe002377e02e8 in term 1.
I20260620 11:42:32.464480 8289 leader_election.cc:302] T a64cfba0d8344082a7fd69dd5ac3c94f P df86ed971c13424aba5fe002377e02e8 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 08b45869e2ab438dba72a0323d9a56bb, df86ed971c13424aba5fe002377e02e8; no voters:
I20260620 11:42:32.464567 8711 raft_consensus.cc:2781] T a64cfba0d8344082a7fd69dd5ac3c94f P df86ed971c13424aba5fe002377e02e8 [term 1 FOLLOWER]: Leader election won for term 1
I20260620 11:42:32.464732 8711 raft_consensus.cc:686] T a64cfba0d8344082a7fd69dd5ac3c94f P df86ed971c13424aba5fe002377e02e8 [term 1 LEADER]: Becoming Leader. State: Replica: df86ed971c13424aba5fe002377e02e8, State: Running, Role: LEADER
I20260620 11:42:32.464829 8711 consensus_queue.cc:237] T a64cfba0d8344082a7fd69dd5ac3c94f P df86ed971c13424aba5fe002377e02e8 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "08b45869e2ab438dba72a0323d9a56bb" member_type: VOTER last_known_addr { host: "127.6.234.194" port: 37745 } } peers { permanent_uuid: "eab50d9c8b4e4cf6afe28767b40bd8f4" member_type: VOTER last_known_addr { host: "127.6.234.195" port: 33411 } } peers { permanent_uuid: "df86ed971c13424aba5fe002377e02e8" member_type: VOTER last_known_addr { host: "127.6.234.193" port: 42909 } }
I20260620 11:42:32.465607 8197 catalog_manager.cc:5495] T a64cfba0d8344082a7fd69dd5ac3c94f P df86ed971c13424aba5fe002377e02e8 reported cstate change: term changed from 0 to 1, leader changed from <none> to df86ed971c13424aba5fe002377e02e8 (127.6.234.193). New cstate: current_term: 1 leader_uuid: "df86ed971c13424aba5fe002377e02e8" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "08b45869e2ab438dba72a0323d9a56bb" member_type: VOTER last_known_addr { host: "127.6.234.194" port: 37745 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "eab50d9c8b4e4cf6afe28767b40bd8f4" member_type: VOTER last_known_addr { host: "127.6.234.195" port: 33411 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "df86ed971c13424aba5fe002377e02e8" member_type: VOTER last_known_addr { host: "127.6.234.193" port: 42909 } health_report { overall_health: HEALTHY } } }
I20260620 11:42:32.479204 8703 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 11:42:32.476510 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34604 (local address 127.6.234.193:42909)
0620 11:42:32.476577 (+ 67us) server_negotiation.cc:206] Beginning negotiation
0620 11:42:32.476579 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 11:42:32.476590 (+ 11us) server_negotiation.cc:407] Connection header received
0620 11:42:32.476667 (+ 77us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 11:42:32.476667 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 11:42:32.476684 (+ 17us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 11:42:32.476739 (+ 55us) server_negotiation.cc:226] Negotiated authn=TOKEN
0620 11:42:32.476903 (+ 164us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 11:42:32.477390 (+ 487us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 11:42:32.478208 (+ 818us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 11:42:32.478409 (+ 201us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 11:42:32.478474 (+ 65us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 11:42:32.478666 (+ 192us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0620 11:42:32.478754 (+ 88us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0620 11:42:32.478814 (+ 60us) server_negotiation.cc:1035] Waiting for connection context
0620 11:42:32.478868 (+ 54us) server_negotiation.cc:299] Negotiation successful
0620 11:42:32.478888 (+ 20us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":46}
I20260620 11:42:32.481029 8635 raft_consensus.cc:1261] T a64cfba0d8344082a7fd69dd5ac3c94f P eab50d9c8b4e4cf6afe28767b40bd8f4 [term 1 FOLLOWER]: Refusing update from remote peer df86ed971c13424aba5fe002377e02e8: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 11:42:32.481033 8495 raft_consensus.cc:1261] T a64cfba0d8344082a7fd69dd5ac3c94f P 08b45869e2ab438dba72a0323d9a56bb [term 1 FOLLOWER]: Refusing update from remote peer df86ed971c13424aba5fe002377e02e8: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 11:42:32.481356 8711 consensus_queue.cc:1035] T a64cfba0d8344082a7fd69dd5ac3c94f P df86ed971c13424aba5fe002377e02e8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "eab50d9c8b4e4cf6afe28767b40bd8f4" member_type: VOTER last_known_addr { host: "127.6.234.195" port: 33411 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 11:42:32.481530 8714 consensus_queue.cc:1035] T a64cfba0d8344082a7fd69dd5ac3c94f P df86ed971c13424aba5fe002377e02e8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "08b45869e2ab438dba72a0323d9a56bb" member_type: VOTER last_known_addr { host: "127.6.234.194" port: 37745 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 11:42:32.484127 8719 mvcc.cc:205] Tried to move back new op lower bound from 7298890762159353856 to 7298890762096549888. Current Snapshot: MvccSnapshot[applied={T|T < 7298890762159353856}]
I20260620 11:42:32.484706 8718 mvcc.cc:205] Tried to move back new op lower bound from 7298890762159353856 to 7298890762096549888. Current Snapshot: MvccSnapshot[applied={T|T < 7298890762159353856 or (T in {7298890762159353856})}]
I20260620 11:42:32.487432 8197 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:46864:
table { table_name: "test-table" } modify_external_catalogs: true
I20260620 11:42:32.487517 8197 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:46864:
table { table_name: "test-table" } modify_external_catalogs: true
I20260620 11:42:32.488313 8197 catalog_manager.cc:5782] T 00000000000000000000000000000000 P a56f5849e11d410fa2127cc9991f437c: Sending DeleteTablet for 3 replicas of tablet a64cfba0d8344082a7fd69dd5ac3c94f
I20260620 11:42:32.488670 8335 tablet_service.cc:1514] Processing DeleteTablet for tablet a64cfba0d8344082a7fd69dd5ac3c94f with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 11:42:32 UTC) from {username='kudu', principal='kudu/127.6.234.254@KRBTEST.COM'} at 127.0.0.1:57570
I20260620 11:42:32.488795 8615 tablet_service.cc:1514] Processing DeleteTablet for tablet a64cfba0d8344082a7fd69dd5ac3c94f with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 11:42:32 UTC) from {username='kudu', principal='kudu/127.6.234.254@KRBTEST.COM'} at 127.0.0.1:49788
I20260620 11:42:32.488790 8475 tablet_service.cc:1514] Processing DeleteTablet for tablet a64cfba0d8344082a7fd69dd5ac3c94f with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 11:42:32 UTC) from {username='kudu', principal='kudu/127.6.234.254@KRBTEST.COM'} at 127.0.0.1:52026
I20260620 11:42:32.488948 8197 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:46864:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260620 11:42:32.489097 8197 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260620 11:42:32.489224 8729 tablet_replica.cc:330] T a64cfba0d8344082a7fd69dd5ac3c94f P 08b45869e2ab438dba72a0323d9a56bb: stopping tablet replica
I20260620 11:42:32.489327 8729 raft_consensus.cc:2229] T a64cfba0d8344082a7fd69dd5ac3c94f P 08b45869e2ab438dba72a0323d9a56bb [term 1 FOLLOWER]: Raft consensus shutting down.
I20260620 11:42:32.489444 8729 raft_consensus.cc:2258] T a64cfba0d8344082a7fd69dd5ac3c94f P 08b45869e2ab438dba72a0323d9a56bb [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 11:42:32.489627 8727 tablet_replica.cc:330] T a64cfba0d8344082a7fd69dd5ac3c94f P df86ed971c13424aba5fe002377e02e8: stopping tablet replica
I20260620 11:42:32.489741 8727 raft_consensus.cc:2229] T a64cfba0d8344082a7fd69dd5ac3c94f P df86ed971c13424aba5fe002377e02e8 [term 1 LEADER]: Raft consensus shutting down.
I20260620 11:42:32.489840 8727 raft_consensus.cc:2258] T a64cfba0d8344082a7fd69dd5ac3c94f P df86ed971c13424aba5fe002377e02e8 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 11:42:32.490047 8728 tablet_replica.cc:330] T a64cfba0d8344082a7fd69dd5ac3c94f P eab50d9c8b4e4cf6afe28767b40bd8f4: stopping tablet replica
I20260620 11:42:32.490140 8729 ts_tablet_manager.cc:1905] T a64cfba0d8344082a7fd69dd5ac3c94f P 08b45869e2ab438dba72a0323d9a56bb: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 11:42:32.490146 8728 raft_consensus.cc:2229] T a64cfba0d8344082a7fd69dd5ac3c94f P eab50d9c8b4e4cf6afe28767b40bd8f4 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260620 11:42:32.490269 8728 raft_consensus.cc:2258] T a64cfba0d8344082a7fd69dd5ac3c94f P eab50d9c8b4e4cf6afe28767b40bd8f4 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 11:42:32.490690 8475 tablet_service.cc:1467] Processing CreateTablet for tablet b2e3e3c341854e3ea73763f4a8326c8d (DEFAULT_TABLE table=test-table [id=c1a0da12220e4e86bee6aa9a8aef4fe3]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 11:42:32.490828 8475 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b2e3e3c341854e3ea73763f4a8326c8d. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 11:42:32.490878 8615 tablet_service.cc:1467] Processing CreateTablet for tablet b2e3e3c341854e3ea73763f4a8326c8d (DEFAULT_TABLE table=test-table [id=c1a0da12220e4e86bee6aa9a8aef4fe3]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 11:42:32.490998 8615 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b2e3e3c341854e3ea73763f4a8326c8d. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 11:42:32.491668 8728 ts_tablet_manager.cc:1905] T a64cfba0d8344082a7fd69dd5ac3c94f P eab50d9c8b4e4cf6afe28767b40bd8f4: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 11:42:32.492249 8729 ts_tablet_manager.cc:1918] T a64cfba0d8344082a7fd69dd5ac3c94f P 08b45869e2ab438dba72a0323d9a56bb: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260620 11:42:32.492275 8335 tablet_service.cc:1467] Processing CreateTablet for tablet b2e3e3c341854e3ea73763f4a8326c8d (DEFAULT_TABLE table=test-table [id=c1a0da12220e4e86bee6aa9a8aef4fe3]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 11:42:32.492362 8704 tablet_bootstrap.cc:492] T b2e3e3c341854e3ea73763f4a8326c8d P eab50d9c8b4e4cf6afe28767b40bd8f4: Bootstrap starting.
I20260620 11:42:32.492388 8335 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b2e3e3c341854e3ea73763f4a8326c8d. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 11:42:32.492403 8729 log.cc:1196] T a64cfba0d8344082a7fd69dd5ac3c94f P 08b45869e2ab438dba72a0323d9a56bb: Deleting WAL directory at /tmp/dist-test-task1JRg7N/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781955746769051-7083-0/minicluster-data/ts-1/wal/wals/a64cfba0d8344082a7fd69dd5ac3c94f
I20260620 11:42:32.492775 8704 tablet_bootstrap.cc:654] T b2e3e3c341854e3ea73763f4a8326c8d P eab50d9c8b4e4cf6afe28767b40bd8f4: Neither blocks nor log segments found. Creating new log.
I20260620 11:42:32.492785 8729 ts_tablet_manager.cc:1939] T a64cfba0d8344082a7fd69dd5ac3c94f P 08b45869e2ab438dba72a0323d9a56bb: Deleting consensus metadata
I20260620 11:42:32.493196 8728 ts_tablet_manager.cc:1918] T a64cfba0d8344082a7fd69dd5ac3c94f P eab50d9c8b4e4cf6afe28767b40bd8f4: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260620 11:42:32.493270 8728 log.cc:1196] T a64cfba0d8344082a7fd69dd5ac3c94f P eab50d9c8b4e4cf6afe28767b40bd8f4: Deleting WAL directory at /tmp/dist-test-task1JRg7N/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781955746769051-7083-0/minicluster-data/ts-2/wal/wals/a64cfba0d8344082a7fd69dd5ac3c94f
I20260620 11:42:32.493503 8704 tablet_bootstrap.cc:492] T b2e3e3c341854e3ea73763f4a8326c8d P eab50d9c8b4e4cf6afe28767b40bd8f4: No bootstrap required, opened a new log
I20260620 11:42:32.493598 8704 ts_tablet_manager.cc:1397] T b2e3e3c341854e3ea73763f4a8326c8d P eab50d9c8b4e4cf6afe28767b40bd8f4: Time spent bootstrapping tablet: real 0.001s user 0.000s sys 0.001s
I20260620 11:42:32.493780 8704 raft_consensus.cc:348] T b2e3e3c341854e3ea73763f4a8326c8d P eab50d9c8b4e4cf6afe28767b40bd8f4 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df86ed971c13424aba5fe002377e02e8" member_type: VOTER last_known_addr { host: "127.6.234.193" port: 42909 } } peers { permanent_uuid: "08b45869e2ab438dba72a0323d9a56bb" member_type: VOTER last_known_addr { host: "127.6.234.194" port: 37745 } } peers { permanent_uuid: "eab50d9c8b4e4cf6afe28767b40bd8f4" member_type: VOTER last_known_addr { host: "127.6.234.195" port: 33411 } }
I20260620 11:42:32.493835 8706 tablet_bootstrap.cc:492] T b2e3e3c341854e3ea73763f4a8326c8d P df86ed971c13424aba5fe002377e02e8: Bootstrap starting.
I20260620 11:42:32.493857 8704 raft_consensus.cc:374] T b2e3e3c341854e3ea73763f4a8326c8d P eab50d9c8b4e4cf6afe28767b40bd8f4 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 11:42:32.493841 8705 tablet_bootstrap.cc:492] T b2e3e3c341854e3ea73763f4a8326c8d P 08b45869e2ab438dba72a0323d9a56bb: Bootstrap starting.
I20260620 11:42:32.493880 8704 raft_consensus.cc:729] T b2e3e3c341854e3ea73763f4a8326c8d P eab50d9c8b4e4cf6afe28767b40bd8f4 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: eab50d9c8b4e4cf6afe28767b40bd8f4, State: Initialized, Role: FOLLOWER
I20260620 11:42:32.493911 8727 ts_tablet_manager.cc:1905] T a64cfba0d8344082a7fd69dd5ac3c94f P df86ed971c13424aba5fe002377e02e8: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 11:42:32.493921 8704 consensus_queue.cc:260] T b2e3e3c341854e3ea73763f4a8326c8d P eab50d9c8b4e4cf6afe28767b40bd8f4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df86ed971c13424aba5fe002377e02e8" member_type: VOTER last_known_addr { host: "127.6.234.193" port: 42909 } } peers { permanent_uuid: "08b45869e2ab438dba72a0323d9a56bb" member_type: VOTER last_known_addr { host: "127.6.234.194" port: 37745 } } peers { permanent_uuid: "eab50d9c8b4e4cf6afe28767b40bd8f4" member_type: VOTER last_known_addr { host: "127.6.234.195" port: 33411 } }
I20260620 11:42:32.494011 8704 ts_tablet_manager.cc:1428] T b2e3e3c341854e3ea73763f4a8326c8d P eab50d9c8b4e4cf6afe28767b40bd8f4: Time spent starting tablet: real 0.000s user 0.000s sys 0.001s
I20260620 11:42:32.494231 8706 tablet_bootstrap.cc:654] T b2e3e3c341854e3ea73763f4a8326c8d P df86ed971c13424aba5fe002377e02e8: Neither blocks nor log segments found. Creating new log.
I20260620 11:42:32.494239 8705 tablet_bootstrap.cc:654] T b2e3e3c341854e3ea73763f4a8326c8d P 08b45869e2ab438dba72a0323d9a56bb: Neither blocks nor log segments found. Creating new log.
I20260620 11:42:32.494621 8185 catalog_manager.cc:4841] TS 08b45869e2ab438dba72a0323d9a56bb (127.6.234.194:37745): tablet a64cfba0d8344082a7fd69dd5ac3c94f (table test-table [id=6fd0eb701fde4a8eb98bb6938552469d]) successfully deleted
I20260620 11:42:32.494956 8706 tablet_bootstrap.cc:492] T b2e3e3c341854e3ea73763f4a8326c8d P df86ed971c13424aba5fe002377e02e8: No bootstrap required, opened a new log
I20260620 11:42:32.495008 8706 ts_tablet_manager.cc:1397] T b2e3e3c341854e3ea73763f4a8326c8d P df86ed971c13424aba5fe002377e02e8: Time spent bootstrapping tablet: real 0.001s user 0.000s sys 0.001s
I20260620 11:42:32.495062 8705 tablet_bootstrap.cc:492] T b2e3e3c341854e3ea73763f4a8326c8d P 08b45869e2ab438dba72a0323d9a56bb: No bootstrap required, opened a new log
I20260620 11:42:32.495110 8705 ts_tablet_manager.cc:1397] T b2e3e3c341854e3ea73763f4a8326c8d P 08b45869e2ab438dba72a0323d9a56bb: Time spent bootstrapping tablet: real 0.001s user 0.001s sys 0.000s
I20260620 11:42:32.495167 8706 raft_consensus.cc:348] T b2e3e3c341854e3ea73763f4a8326c8d P df86ed971c13424aba5fe002377e02e8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df86ed971c13424aba5fe002377e02e8" member_type: VOTER last_known_addr { host: "127.6.234.193" port: 42909 } } peers { permanent_uuid: "08b45869e2ab438dba72a0323d9a56bb" member_type: VOTER last_known_addr { host: "127.6.234.194" port: 37745 } } peers { permanent_uuid: "eab50d9c8b4e4cf6afe28767b40bd8f4" member_type: VOTER last_known_addr { host: "127.6.234.195" port: 33411 } }
I20260620 11:42:32.495254 8706 raft_consensus.cc:374] T b2e3e3c341854e3ea73763f4a8326c8d P df86ed971c13424aba5fe002377e02e8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 11:42:32.495283 8706 raft_consensus.cc:729] T b2e3e3c341854e3ea73763f4a8326c8d P df86ed971c13424aba5fe002377e02e8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: df86ed971c13424aba5fe002377e02e8, State: Initialized, Role: FOLLOWER
I20260620 11:42:32.495262 8705 raft_consensus.cc:348] T b2e3e3c341854e3ea73763f4a8326c8d P 08b45869e2ab438dba72a0323d9a56bb [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df86ed971c13424aba5fe002377e02e8" member_type: VOTER last_known_addr { host: "127.6.234.193" port: 42909 } } peers { permanent_uuid: "08b45869e2ab438dba72a0323d9a56bb" member_type: VOTER last_known_addr { host: "127.6.234.194" port: 37745 } } peers { permanent_uuid: "eab50d9c8b4e4cf6afe28767b40bd8f4" member_type: VOTER last_known_addr { host: "127.6.234.195" port: 33411 } }
I20260620 11:42:32.495342 8705 raft_consensus.cc:374] T b2e3e3c341854e3ea73763f4a8326c8d P 08b45869e2ab438dba72a0323d9a56bb [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 11:42:32.495334 8706 consensus_queue.cc:260] T b2e3e3c341854e3ea73763f4a8326c8d P df86ed971c13424aba5fe002377e02e8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df86ed971c13424aba5fe002377e02e8" member_type: VOTER last_known_addr { host: "127.6.234.193" port: 42909 } } peers { permanent_uuid: "08b45869e2ab438dba72a0323d9a56bb" member_type: VOTER last_known_addr { host: "127.6.234.194" port: 37745 } } peers { permanent_uuid: "eab50d9c8b4e4cf6afe28767b40bd8f4" member_type: VOTER last_known_addr { host: "127.6.234.195" port: 33411 } }
I20260620 11:42:32.495368 8705 raft_consensus.cc:729] T b2e3e3c341854e3ea73763f4a8326c8d P 08b45869e2ab438dba72a0323d9a56bb [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 08b45869e2ab438dba72a0323d9a56bb, State: Initialized, Role: FOLLOWER
I20260620 11:42:32.495371 8727 ts_tablet_manager.cc:1918] T a64cfba0d8344082a7fd69dd5ac3c94f P df86ed971c13424aba5fe002377e02e8: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260620 11:42:32.495410 8705 consensus_queue.cc:260] T b2e3e3c341854e3ea73763f4a8326c8d P 08b45869e2ab438dba72a0323d9a56bb [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df86ed971c13424aba5fe002377e02e8" member_type: VOTER last_known_addr { host: "127.6.234.193" port: 42909 } } peers { permanent_uuid: "08b45869e2ab438dba72a0323d9a56bb" member_type: VOTER last_known_addr { host: "127.6.234.194" port: 37745 } } peers { permanent_uuid: "eab50d9c8b4e4cf6afe28767b40bd8f4" member_type: VOTER last_known_addr { host: "127.6.234.195" port: 33411 } }
I20260620 11:42:32.495422 8706 ts_tablet_manager.cc:1428] T b2e3e3c341854e3ea73763f4a8326c8d P df86ed971c13424aba5fe002377e02e8: Time spent starting tablet: real 0.000s user 0.000s sys 0.001s
I20260620 11:42:32.495504 8705 ts_tablet_manager.cc:1428] T b2e3e3c341854e3ea73763f4a8326c8d P 08b45869e2ab438dba72a0323d9a56bb: Time spent starting tablet: real 0.000s user 0.001s sys 0.000s
I20260620 11:42:32.495434 8727 log.cc:1196] T a64cfba0d8344082a7fd69dd5ac3c94f P df86ed971c13424aba5fe002377e02e8: Deleting WAL directory at /tmp/dist-test-task1JRg7N/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781955746769051-7083-0/minicluster-data/ts-0/wal/wals/a64cfba0d8344082a7fd69dd5ac3c94f
I20260620 11:42:32.495857 8727 ts_tablet_manager.cc:1939] T a64cfba0d8344082a7fd69dd5ac3c94f P df86ed971c13424aba5fe002377e02e8: Deleting consensus metadata
I20260620 11:42:32.493518 8728 ts_tablet_manager.cc:1939] T a64cfba0d8344082a7fd69dd5ac3c94f P eab50d9c8b4e4cf6afe28767b40bd8f4: Deleting consensus metadata
I20260620 11:42:32.496220 8182 catalog_manager.cc:4841] TS df86ed971c13424aba5fe002377e02e8 (127.6.234.193:42909): tablet a64cfba0d8344082a7fd69dd5ac3c94f (table test-table [id=6fd0eb701fde4a8eb98bb6938552469d]) successfully deleted
I20260620 11:42:32.496503 8185 catalog_manager.cc:4841] TS eab50d9c8b4e4cf6afe28767b40bd8f4 (127.6.234.195:33411): tablet a64cfba0d8344082a7fd69dd5ac3c94f (table test-table [id=6fd0eb701fde4a8eb98bb6938552469d]) successfully deleted
W20260620 11:42:32.544008 8402 tablet.cc:2369] T b2e3e3c341854e3ea73763f4a8326c8d P df86ed971c13424aba5fe002377e02e8: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260620 11:42:32.637458 8682 tablet.cc:2369] T b2e3e3c341854e3ea73763f4a8326c8d P eab50d9c8b4e4cf6afe28767b40bd8f4: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260620 11:42:32.720710 8714 raft_consensus.cc:482] T b2e3e3c341854e3ea73763f4a8326c8d P df86ed971c13424aba5fe002377e02e8 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260620 11:42:32.720829 8714 raft_consensus.cc:504] T b2e3e3c341854e3ea73763f4a8326c8d P df86ed971c13424aba5fe002377e02e8 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df86ed971c13424aba5fe002377e02e8" member_type: VOTER last_known_addr { host: "127.6.234.193" port: 42909 } } peers { permanent_uuid: "08b45869e2ab438dba72a0323d9a56bb" member_type: VOTER last_known_addr { host: "127.6.234.194" port: 37745 } } peers { permanent_uuid: "eab50d9c8b4e4cf6afe28767b40bd8f4" member_type: VOTER last_known_addr { host: "127.6.234.195" port: 33411 } }
I20260620 11:42:32.721021 8714 leader_election.cc:288] T b2e3e3c341854e3ea73763f4a8326c8d P df86ed971c13424aba5fe002377e02e8 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 08b45869e2ab438dba72a0323d9a56bb (127.6.234.194:37745), eab50d9c8b4e4cf6afe28767b40bd8f4 (127.6.234.195:33411)
I20260620 11:42:32.721254 8495 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b2e3e3c341854e3ea73763f4a8326c8d" candidate_uuid: "df86ed971c13424aba5fe002377e02e8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "08b45869e2ab438dba72a0323d9a56bb" is_pre_election: true
I20260620 11:42:32.721355 8495 raft_consensus.cc:2454] T b2e3e3c341854e3ea73763f4a8326c8d P 08b45869e2ab438dba72a0323d9a56bb [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate df86ed971c13424aba5fe002377e02e8 in term 0.
I20260620 11:42:32.721323 8635 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b2e3e3c341854e3ea73763f4a8326c8d" candidate_uuid: "df86ed971c13424aba5fe002377e02e8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "eab50d9c8b4e4cf6afe28767b40bd8f4" is_pre_election: true
I20260620 11:42:32.721470 8635 raft_consensus.cc:2454] T b2e3e3c341854e3ea73763f4a8326c8d P eab50d9c8b4e4cf6afe28767b40bd8f4 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate df86ed971c13424aba5fe002377e02e8 in term 0.
I20260620 11:42:32.721508 8289 leader_election.cc:302] T b2e3e3c341854e3ea73763f4a8326c8d P df86ed971c13424aba5fe002377e02e8 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 08b45869e2ab438dba72a0323d9a56bb, df86ed971c13424aba5fe002377e02e8; no voters:
I20260620 11:42:32.721644 8714 raft_consensus.cc:2781] T b2e3e3c341854e3ea73763f4a8326c8d P df86ed971c13424aba5fe002377e02e8 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260620 11:42:32.721697 8714 raft_consensus.cc:482] T b2e3e3c341854e3ea73763f4a8326c8d P df86ed971c13424aba5fe002377e02e8 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260620 11:42:32.721750 8714 raft_consensus.cc:3037] T b2e3e3c341854e3ea73763f4a8326c8d P df86ed971c13424aba5fe002377e02e8 [term 0 FOLLOWER]: Advancing to term 1
I20260620 11:42:32.722455 8714 raft_consensus.cc:504] T b2e3e3c341854e3ea73763f4a8326c8d P df86ed971c13424aba5fe002377e02e8 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df86ed971c13424aba5fe002377e02e8" member_type: VOTER last_known_addr { host: "127.6.234.193" port: 42909 } } peers { permanent_uuid: "08b45869e2ab438dba72a0323d9a56bb" member_type: VOTER last_known_addr { host: "127.6.234.194" port: 37745 } } peers { permanent_uuid: "eab50d9c8b4e4cf6afe28767b40bd8f4" member_type: VOTER last_known_addr { host: "127.6.234.195" port: 33411 } }
I20260620 11:42:32.722596 8714 leader_election.cc:288] T b2e3e3c341854e3ea73763f4a8326c8d P df86ed971c13424aba5fe002377e02e8 [CANDIDATE]: Term 1 election: Requested vote from peers 08b45869e2ab438dba72a0323d9a56bb (127.6.234.194:37745), eab50d9c8b4e4cf6afe28767b40bd8f4 (127.6.234.195:33411)
I20260620 11:42:32.722774 8635 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b2e3e3c341854e3ea73763f4a8326c8d" candidate_uuid: "df86ed971c13424aba5fe002377e02e8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "eab50d9c8b4e4cf6afe28767b40bd8f4"
I20260620 11:42:32.722772 8495 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b2e3e3c341854e3ea73763f4a8326c8d" candidate_uuid: "df86ed971c13424aba5fe002377e02e8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "08b45869e2ab438dba72a0323d9a56bb"
I20260620 11:42:32.722852 8635 raft_consensus.cc:3037] T b2e3e3c341854e3ea73763f4a8326c8d P eab50d9c8b4e4cf6afe28767b40bd8f4 [term 0 FOLLOWER]: Advancing to term 1
I20260620 11:42:32.722852 8495 raft_consensus.cc:3037] T b2e3e3c341854e3ea73763f4a8326c8d P 08b45869e2ab438dba72a0323d9a56bb [term 0 FOLLOWER]: Advancing to term 1
I20260620 11:42:32.723596 8635 raft_consensus.cc:2454] T b2e3e3c341854e3ea73763f4a8326c8d P eab50d9c8b4e4cf6afe28767b40bd8f4 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate df86ed971c13424aba5fe002377e02e8 in term 1.
I20260620 11:42:32.723596 8495 raft_consensus.cc:2454] T b2e3e3c341854e3ea73763f4a8326c8d P 08b45869e2ab438dba72a0323d9a56bb [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate df86ed971c13424aba5fe002377e02e8 in term 1.
I20260620 11:42:32.723788 8289 leader_election.cc:302] T b2e3e3c341854e3ea73763f4a8326c8d P df86ed971c13424aba5fe002377e02e8 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: df86ed971c13424aba5fe002377e02e8, eab50d9c8b4e4cf6afe28767b40bd8f4; no voters:
I20260620 11:42:32.723865 8714 raft_consensus.cc:2781] T b2e3e3c341854e3ea73763f4a8326c8d P df86ed971c13424aba5fe002377e02e8 [term 1 FOLLOWER]: Leader election won for term 1
I20260620 11:42:32.723930 8714 raft_consensus.cc:686] T b2e3e3c341854e3ea73763f4a8326c8d P df86ed971c13424aba5fe002377e02e8 [term 1 LEADER]: Becoming Leader. State: Replica: df86ed971c13424aba5fe002377e02e8, State: Running, Role: LEADER
I20260620 11:42:32.723980 8714 consensus_queue.cc:237] T b2e3e3c341854e3ea73763f4a8326c8d P df86ed971c13424aba5fe002377e02e8 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df86ed971c13424aba5fe002377e02e8" member_type: VOTER last_known_addr { host: "127.6.234.193" port: 42909 } } peers { permanent_uuid: "08b45869e2ab438dba72a0323d9a56bb" member_type: VOTER last_known_addr { host: "127.6.234.194" port: 37745 } } peers { permanent_uuid: "eab50d9c8b4e4cf6afe28767b40bd8f4" member_type: VOTER last_known_addr { host: "127.6.234.195" port: 33411 } }
I20260620 11:42:32.724525 8198 catalog_manager.cc:5495] T b2e3e3c341854e3ea73763f4a8326c8d P df86ed971c13424aba5fe002377e02e8 reported cstate change: term changed from 0 to 1, leader changed from <none> to df86ed971c13424aba5fe002377e02e8 (127.6.234.193). New cstate: current_term: 1 leader_uuid: "df86ed971c13424aba5fe002377e02e8" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df86ed971c13424aba5fe002377e02e8" member_type: VOTER last_known_addr { host: "127.6.234.193" port: 42909 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "08b45869e2ab438dba72a0323d9a56bb" member_type: VOTER last_known_addr { host: "127.6.234.194" port: 37745 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "eab50d9c8b4e4cf6afe28767b40bd8f4" member_type: VOTER last_known_addr { host: "127.6.234.195" port: 33411 } health_report { overall_health: UNKNOWN } } }
Jun 20 11:42:32 dist-test-slave-cjc6 krb5kdc[8153](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781955751, etypes {rep=17 tkt=17 ses=17}, kudu/127.6.234.254@KRBTEST.COM for kudu/127.6.234.254@KRBTEST.COM
I20260620 11:42:32.809299 8739 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 11:42:32.802528 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.234.254:43185 (local address 127.0.0.1:41780)
0620 11:42:32.802754 (+ 226us) negotiation.cc:107] Waiting for socket to connect
0620 11:42:32.802763 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0620 11:42:32.802821 (+ 58us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 11:42:32.803014 (+ 193us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 11:42:32.803015 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 11:42:32.803416 (+ 401us) client_negotiation.cc:189] Negotiated authn=SASL
0620 11:42:32.803471 (+ 55us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 11:42:32.803474 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 11:42:32.804142 (+ 668us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 11:42:32.804143 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 11:42:32.804707 (+ 564us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 11:42:32.804709 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 11:42:32.805065 (+ 356us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 11:42:32.805066 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 11:42:32.805115 (+ 49us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 11:42:32.805600 (+ 485us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 11:42:32.805604 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0620 11:42:32.807032 (+ 1428us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 11:42:32.808383 (+ 1351us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 11:42:32.808390 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 11:42:32.808391 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0620 11:42:32.808590 (+ 199us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 11:42:32.808763 (+ 173us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 11:42:32.808764 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 11:42:32.808764 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0620 11:42:32.808819 (+ 55us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 11:42:32.809021 (+ 202us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 11:42:32.809026 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 11:42:32.809068 (+ 42us) client_negotiation.cc:769] Sending connection context
0620 11:42:32.809119 (+ 51us) client_negotiation.cc:240] Negotiation successful
0620 11:42:32.809205 (+ 86us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":203,"thread_start_us":112,"threads_started":1}
I20260620 11:42:32.809346 8248 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 11:42:32.802619 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41780 (local address 127.6.234.254:43185)
0620 11:42:32.802706 (+ 87us) server_negotiation.cc:206] Beginning negotiation
0620 11:42:32.802708 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 11:42:32.802823 (+ 115us) server_negotiation.cc:407] Connection header received
0620 11:42:32.802869 (+ 46us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 11:42:32.802870 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 11:42:32.802881 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 11:42:32.802998 (+ 117us) server_negotiation.cc:226] Negotiated authn=SASL
0620 11:42:32.803568 (+ 570us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 11:42:32.804059 (+ 491us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 11:42:32.804803 (+ 744us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 11:42:32.804983 (+ 180us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 11:42:32.805064 (+ 81us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 11:42:32.807128 (+ 2064us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 11:42:32.807129 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 11:42:32.807129 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 11:42:32.807135 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0620 11:42:32.808298 (+ 1163us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 11:42:32.808657 (+ 359us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 11:42:32.808658 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 11:42:32.808658 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 11:42:32.808705 (+ 47us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 11:42:32.808871 (+ 166us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 11:42:32.808871 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 11:42:32.808872 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 11:42:32.808969 (+ 97us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 11:42:32.809016 (+ 47us) server_negotiation.cc:1035] Waiting for connection context
0620 11:42:32.809151 (+ 135us) server_negotiation.cc:299] Negotiation successful
0620 11:42:32.809249 (+ 98us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":36,"server-negotiator.queue_time_us":52}
I20260620 11:42:32.811429 8198 catalog_manager.cc:1908] Servicing CreateTable request from {username='kudu', principal='kudu/127.6.234.254@KRBTEST.COM'} at 127.0.0.1:41780:
name: "kudu_system.kudu_transactions"
schema {
columns {
name: "txn_id"
type: INT64
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "entry_type"
type: INT8
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "identifier"
type: STRING
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "metadata"
type: STRING
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
rows: "<redacted>""\006\001\000\000\000\000\000\000\000\000\007\001@B\017\000\000\000\000\000"
indirect_data: "<redacted>"""
}
partition_schema {
range_schema {
columns {
name: "txn_id"
}
}
}
table_type: TXN_STATUS_TABLE
W20260620 11:42:32.811655 8198 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table kudu_system.kudu_transactions in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260620 11:42:32.813042 8335 tablet_service.cc:1467] Processing CreateTablet for tablet 2308e22bab7d486c95224c056c7e72b4 (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=a3aee160014448d2b68ebfc069272ee9]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260620 11:42:32.813077 8475 tablet_service.cc:1467] Processing CreateTablet for tablet 2308e22bab7d486c95224c056c7e72b4 (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=a3aee160014448d2b68ebfc069272ee9]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260620 11:42:32.813102 8615 tablet_service.cc:1467] Processing CreateTablet for tablet 2308e22bab7d486c95224c056c7e72b4 (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=a3aee160014448d2b68ebfc069272ee9]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260620 11:42:32.813221 8475 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 2308e22bab7d486c95224c056c7e72b4. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 11:42:32.813221 8335 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 2308e22bab7d486c95224c056c7e72b4. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 11:42:32.813283 8615 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 2308e22bab7d486c95224c056c7e72b4. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 11:42:32.814718 8706 tablet_bootstrap.cc:492] T 2308e22bab7d486c95224c056c7e72b4 P df86ed971c13424aba5fe002377e02e8: Bootstrap starting.
I20260620 11:42:32.814723 8705 tablet_bootstrap.cc:492] T 2308e22bab7d486c95224c056c7e72b4 P 08b45869e2ab438dba72a0323d9a56bb: Bootstrap starting.
I20260620 11:42:32.814767 8704 tablet_bootstrap.cc:492] T 2308e22bab7d486c95224c056c7e72b4 P eab50d9c8b4e4cf6afe28767b40bd8f4: Bootstrap starting.
I20260620 11:42:32.815034 8706 tablet_bootstrap.cc:654] T 2308e22bab7d486c95224c056c7e72b4 P df86ed971c13424aba5fe002377e02e8: Neither blocks nor log segments found. Creating new log.
I20260620 11:42:32.815161 8704 tablet_bootstrap.cc:654] T 2308e22bab7d486c95224c056c7e72b4 P eab50d9c8b4e4cf6afe28767b40bd8f4: Neither blocks nor log segments found. Creating new log.
I20260620 11:42:32.815161 8705 tablet_bootstrap.cc:654] T 2308e22bab7d486c95224c056c7e72b4 P 08b45869e2ab438dba72a0323d9a56bb: Neither blocks nor log segments found. Creating new log.
I20260620 11:42:32.815614 8706 tablet_bootstrap.cc:492] T 2308e22bab7d486c95224c056c7e72b4 P df86ed971c13424aba5fe002377e02e8: No bootstrap required, opened a new log
I20260620 11:42:32.815672 8706 ts_tablet_manager.cc:1397] T 2308e22bab7d486c95224c056c7e72b4 P df86ed971c13424aba5fe002377e02e8: Time spent bootstrapping tablet: real 0.001s user 0.000s sys 0.001s
I20260620 11:42:32.815793 8704 tablet_bootstrap.cc:492] T 2308e22bab7d486c95224c056c7e72b4 P eab50d9c8b4e4cf6afe28767b40bd8f4: No bootstrap required, opened a new log
I20260620 11:42:32.815801 8706 raft_consensus.cc:348] T 2308e22bab7d486c95224c056c7e72b4 P df86ed971c13424aba5fe002377e02e8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df86ed971c13424aba5fe002377e02e8" member_type: VOTER last_known_addr { host: "127.6.234.193" port: 42909 } } peers { permanent_uuid: "08b45869e2ab438dba72a0323d9a56bb" member_type: VOTER last_known_addr { host: "127.6.234.194" port: 37745 } } peers { permanent_uuid: "eab50d9c8b4e4cf6afe28767b40bd8f4" member_type: VOTER last_known_addr { host: "127.6.234.195" port: 33411 } }
I20260620 11:42:32.815868 8706 raft_consensus.cc:374] T 2308e22bab7d486c95224c056c7e72b4 P df86ed971c13424aba5fe002377e02e8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 11:42:32.815865 8704 ts_tablet_manager.cc:1397] T 2308e22bab7d486c95224c056c7e72b4 P eab50d9c8b4e4cf6afe28767b40bd8f4: Time spent bootstrapping tablet: real 0.001s user 0.000s sys 0.001s
I20260620 11:42:32.815896 8706 raft_consensus.cc:729] T 2308e22bab7d486c95224c056c7e72b4 P df86ed971c13424aba5fe002377e02e8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: df86ed971c13424aba5fe002377e02e8, State: Initialized, Role: FOLLOWER
I20260620 11:42:32.815955 8706 consensus_queue.cc:260] T 2308e22bab7d486c95224c056c7e72b4 P df86ed971c13424aba5fe002377e02e8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df86ed971c13424aba5fe002377e02e8" member_type: VOTER last_known_addr { host: "127.6.234.193" port: 42909 } } peers { permanent_uuid: "08b45869e2ab438dba72a0323d9a56bb" member_type: VOTER last_known_addr { host: "127.6.234.194" port: 37745 } } peers { permanent_uuid: "eab50d9c8b4e4cf6afe28767b40bd8f4" member_type: VOTER last_known_addr { host: "127.6.234.195" port: 33411 } }
I20260620 11:42:32.815989 8705 tablet_bootstrap.cc:492] T 2308e22bab7d486c95224c056c7e72b4 P 08b45869e2ab438dba72a0323d9a56bb: No bootstrap required, opened a new log
I20260620 11:42:32.816030 8705 ts_tablet_manager.cc:1397] T 2308e22bab7d486c95224c056c7e72b4 P 08b45869e2ab438dba72a0323d9a56bb: Time spent bootstrapping tablet: real 0.001s user 0.001s sys 0.000s
I20260620 11:42:32.816047 8706 ts_tablet_manager.cc:1428] T 2308e22bab7d486c95224c056c7e72b4 P df86ed971c13424aba5fe002377e02e8: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
I20260620 11:42:32.815999 8704 raft_consensus.cc:348] T 2308e22bab7d486c95224c056c7e72b4 P eab50d9c8b4e4cf6afe28767b40bd8f4 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df86ed971c13424aba5fe002377e02e8" member_type: VOTER last_known_addr { host: "127.6.234.193" port: 42909 } } peers { permanent_uuid: "08b45869e2ab438dba72a0323d9a56bb" member_type: VOTER last_known_addr { host: "127.6.234.194" port: 37745 } } peers { permanent_uuid: "eab50d9c8b4e4cf6afe28767b40bd8f4" member_type: VOTER last_known_addr { host: "127.6.234.195" port: 33411 } }
I20260620 11:42:32.816080 8704 raft_consensus.cc:374] T 2308e22bab7d486c95224c056c7e72b4 P eab50d9c8b4e4cf6afe28767b40bd8f4 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 11:42:32.816107 8704 raft_consensus.cc:729] T 2308e22bab7d486c95224c056c7e72b4 P eab50d9c8b4e4cf6afe28767b40bd8f4 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: eab50d9c8b4e4cf6afe28767b40bd8f4, State: Initialized, Role: FOLLOWER
I20260620 11:42:32.816069 8714 tablet_replica.cc:439] T 2308e22bab7d486c95224c056c7e72b4 P df86ed971c13424aba5fe002377e02e8: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df86ed971c13424aba5fe002377e02e8" member_type: VOTER last_known_addr { host: "127.6.234.193" port: 42909 } } peers { permanent_uuid: "08b45869e2ab438dba72a0323d9a56bb" member_type: VOTER last_known_addr { host: "127.6.234.194" port: 37745 } } peers { permanent_uuid: "eab50d9c8b4e4cf6afe28767b40bd8f4" member_type: VOTER last_known_addr { host: "127.6.234.195" port: 33411 } } }
I20260620 11:42:32.816140 8714 tablet_replica.cc:442] T 2308e22bab7d486c95224c056c7e72b4 P df86ed971c13424aba5fe002377e02e8: This TxnStatusTablet replica's current role is: FOLLOWER
I20260620 11:42:32.816155 8704 consensus_queue.cc:260] T 2308e22bab7d486c95224c056c7e72b4 P eab50d9c8b4e4cf6afe28767b40bd8f4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df86ed971c13424aba5fe002377e02e8" member_type: VOTER last_known_addr { host: "127.6.234.193" port: 42909 } } peers { permanent_uuid: "08b45869e2ab438dba72a0323d9a56bb" member_type: VOTER last_known_addr { host: "127.6.234.194" port: 37745 } } peers { permanent_uuid: "eab50d9c8b4e4cf6afe28767b40bd8f4" member_type: VOTER last_known_addr { host: "127.6.234.195" port: 33411 } }
I20260620 11:42:32.816174 8705 raft_consensus.cc:348] T 2308e22bab7d486c95224c056c7e72b4 P 08b45869e2ab438dba72a0323d9a56bb [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df86ed971c13424aba5fe002377e02e8" member_type: VOTER last_known_addr { host: "127.6.234.193" port: 42909 } } peers { permanent_uuid: "08b45869e2ab438dba72a0323d9a56bb" member_type: VOTER last_known_addr { host: "127.6.234.194" port: 37745 } } peers { permanent_uuid: "eab50d9c8b4e4cf6afe28767b40bd8f4" member_type: VOTER last_known_addr { host: "127.6.234.195" port: 33411 } }
I20260620 11:42:32.816233 8705 raft_consensus.cc:374] T 2308e22bab7d486c95224c056c7e72b4 P 08b45869e2ab438dba72a0323d9a56bb [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 11:42:32.816246 8704 ts_tablet_manager.cc:1428] T 2308e22bab7d486c95224c056c7e72b4 P eab50d9c8b4e4cf6afe28767b40bd8f4: Time spent starting tablet: real 0.000s user 0.000s sys 0.001s
I20260620 11:42:32.816256 8705 raft_consensus.cc:729] T 2308e22bab7d486c95224c056c7e72b4 P 08b45869e2ab438dba72a0323d9a56bb [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 08b45869e2ab438dba72a0323d9a56bb, State: Initialized, Role: FOLLOWER
I20260620 11:42:32.816305 8705 consensus_queue.cc:260] T 2308e22bab7d486c95224c056c7e72b4 P 08b45869e2ab438dba72a0323d9a56bb [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df86ed971c13424aba5fe002377e02e8" member_type: VOTER last_known_addr { host: "127.6.234.193" port: 42909 } } peers { permanent_uuid: "08b45869e2ab438dba72a0323d9a56bb" member_type: VOTER last_known_addr { host: "127.6.234.194" port: 37745 } } peers { permanent_uuid: "eab50d9c8b4e4cf6afe28767b40bd8f4" member_type: VOTER last_known_addr { host: "127.6.234.195" port: 33411 } }
I20260620 11:42:32.816399 8705 ts_tablet_manager.cc:1428] T 2308e22bab7d486c95224c056c7e72b4 P 08b45869e2ab438dba72a0323d9a56bb: Time spent starting tablet: real 0.000s user 0.001s sys 0.000s
I20260620 11:42:32.816395 8710 tablet_replica.cc:439] T 2308e22bab7d486c95224c056c7e72b4 P eab50d9c8b4e4cf6afe28767b40bd8f4: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df86ed971c13424aba5fe002377e02e8" member_type: VOTER last_known_addr { host: "127.6.234.193" port: 42909 } } peers { permanent_uuid: "08b45869e2ab438dba72a0323d9a56bb" member_type: VOTER last_known_addr { host: "127.6.234.194" port: 37745 } } peers { permanent_uuid: "eab50d9c8b4e4cf6afe28767b40bd8f4" member_type: VOTER last_known_addr { host: "127.6.234.195" port: 33411 } } }
I20260620 11:42:32.816471 8710 tablet_replica.cc:442] T 2308e22bab7d486c95224c056c7e72b4 P eab50d9c8b4e4cf6afe28767b40bd8f4: This TxnStatusTablet replica's current role is: FOLLOWER
I20260620 11:42:32.816450 8712 tablet_replica.cc:439] T 2308e22bab7d486c95224c056c7e72b4 P 08b45869e2ab438dba72a0323d9a56bb: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df86ed971c13424aba5fe002377e02e8" member_type: VOTER last_known_addr { host: "127.6.234.193" port: 42909 } } peers { permanent_uuid: "08b45869e2ab438dba72a0323d9a56bb" member_type: VOTER last_known_addr { host: "127.6.234.194" port: 37745 } } peers { permanent_uuid: "eab50d9c8b4e4cf6afe28767b40bd8f4" member_type: VOTER last_known_addr { host: "127.6.234.195" port: 33411 } } }
I20260620 11:42:32.816573 8712 tablet_replica.cc:442] T 2308e22bab7d486c95224c056c7e72b4 P 08b45869e2ab438dba72a0323d9a56bb: This TxnStatusTablet replica's current role is: FOLLOWER
I20260620 11:42:32.942951 8714 raft_consensus.cc:482] T 2308e22bab7d486c95224c056c7e72b4 P df86ed971c13424aba5fe002377e02e8 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260620 11:42:32.943114 8714 raft_consensus.cc:504] T 2308e22bab7d486c95224c056c7e72b4 P df86ed971c13424aba5fe002377e02e8 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df86ed971c13424aba5fe002377e02e8" member_type: VOTER last_known_addr { host: "127.6.234.193" port: 42909 } } peers { permanent_uuid: "08b45869e2ab438dba72a0323d9a56bb" member_type: VOTER last_known_addr { host: "127.6.234.194" port: 37745 } } peers { permanent_uuid: "eab50d9c8b4e4cf6afe28767b40bd8f4" member_type: VOTER last_known_addr { host: "127.6.234.195" port: 33411 } }
I20260620 11:42:32.943266 8714 leader_election.cc:288] T 2308e22bab7d486c95224c056c7e72b4 P df86ed971c13424aba5fe002377e02e8 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 08b45869e2ab438dba72a0323d9a56bb (127.6.234.194:37745), eab50d9c8b4e4cf6afe28767b40bd8f4 (127.6.234.195:33411)
I20260620 11:42:32.943511 8495 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2308e22bab7d486c95224c056c7e72b4" candidate_uuid: "df86ed971c13424aba5fe002377e02e8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "08b45869e2ab438dba72a0323d9a56bb" is_pre_election: true
I20260620 11:42:32.943529 8635 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2308e22bab7d486c95224c056c7e72b4" candidate_uuid: "df86ed971c13424aba5fe002377e02e8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "eab50d9c8b4e4cf6afe28767b40bd8f4" is_pre_election: true
I20260620 11:42:32.943632 8495 raft_consensus.cc:2454] T 2308e22bab7d486c95224c056c7e72b4 P 08b45869e2ab438dba72a0323d9a56bb [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate df86ed971c13424aba5fe002377e02e8 in term 0.
I20260620 11:42:32.943629 8635 raft_consensus.cc:2454] T 2308e22bab7d486c95224c056c7e72b4 P eab50d9c8b4e4cf6afe28767b40bd8f4 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate df86ed971c13424aba5fe002377e02e8 in term 0.
I20260620 11:42:32.943816 8289 leader_election.cc:302] T 2308e22bab7d486c95224c056c7e72b4 P df86ed971c13424aba5fe002377e02e8 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 08b45869e2ab438dba72a0323d9a56bb, df86ed971c13424aba5fe002377e02e8; no voters:
I20260620 11:42:32.943951 8714 raft_consensus.cc:2781] T 2308e22bab7d486c95224c056c7e72b4 P df86ed971c13424aba5fe002377e02e8 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260620 11:42:32.944015 8714 raft_consensus.cc:482] T 2308e22bab7d486c95224c056c7e72b4 P df86ed971c13424aba5fe002377e02e8 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260620 11:42:32.944048 8714 raft_consensus.cc:3037] T 2308e22bab7d486c95224c056c7e72b4 P df86ed971c13424aba5fe002377e02e8 [term 0 FOLLOWER]: Advancing to term 1
I20260620 11:42:32.944825 8714 raft_consensus.cc:504] T 2308e22bab7d486c95224c056c7e72b4 P df86ed971c13424aba5fe002377e02e8 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df86ed971c13424aba5fe002377e02e8" member_type: VOTER last_known_addr { host: "127.6.234.193" port: 42909 } } peers { permanent_uuid: "08b45869e2ab438dba72a0323d9a56bb" member_type: VOTER last_known_addr { host: "127.6.234.194" port: 37745 } } peers { permanent_uuid: "eab50d9c8b4e4cf6afe28767b40bd8f4" member_type: VOTER last_known_addr { host: "127.6.234.195" port: 33411 } }
I20260620 11:42:32.944959 8714 leader_election.cc:288] T 2308e22bab7d486c95224c056c7e72b4 P df86ed971c13424aba5fe002377e02e8 [CANDIDATE]: Term 1 election: Requested vote from peers 08b45869e2ab438dba72a0323d9a56bb (127.6.234.194:37745), eab50d9c8b4e4cf6afe28767b40bd8f4 (127.6.234.195:33411)
I20260620 11:42:32.945191 8495 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2308e22bab7d486c95224c056c7e72b4" candidate_uuid: "df86ed971c13424aba5fe002377e02e8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "08b45869e2ab438dba72a0323d9a56bb"
I20260620 11:42:32.945170 8635 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2308e22bab7d486c95224c056c7e72b4" candidate_uuid: "df86ed971c13424aba5fe002377e02e8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "eab50d9c8b4e4cf6afe28767b40bd8f4"
I20260620 11:42:32.945274 8495 raft_consensus.cc:3037] T 2308e22bab7d486c95224c056c7e72b4 P 08b45869e2ab438dba72a0323d9a56bb [term 0 FOLLOWER]: Advancing to term 1
I20260620 11:42:32.945281 8635 raft_consensus.cc:3037] T 2308e22bab7d486c95224c056c7e72b4 P eab50d9c8b4e4cf6afe28767b40bd8f4 [term 0 FOLLOWER]: Advancing to term 1
I20260620 11:42:32.945983 8495 raft_consensus.cc:2454] T 2308e22bab7d486c95224c056c7e72b4 P 08b45869e2ab438dba72a0323d9a56bb [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate df86ed971c13424aba5fe002377e02e8 in term 1.
I20260620 11:42:32.945968 8635 raft_consensus.cc:2454] T 2308e22bab7d486c95224c056c7e72b4 P eab50d9c8b4e4cf6afe28767b40bd8f4 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate df86ed971c13424aba5fe002377e02e8 in term 1.
I20260620 11:42:32.946142 8289 leader_election.cc:302] T 2308e22bab7d486c95224c056c7e72b4 P df86ed971c13424aba5fe002377e02e8 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 08b45869e2ab438dba72a0323d9a56bb, df86ed971c13424aba5fe002377e02e8; no voters:
I20260620 11:42:32.946239 8714 raft_consensus.cc:2781] T 2308e22bab7d486c95224c056c7e72b4 P df86ed971c13424aba5fe002377e02e8 [term 1 FOLLOWER]: Leader election won for term 1
I20260620 11:42:32.946316 8714 raft_consensus.cc:686] T 2308e22bab7d486c95224c056c7e72b4 P df86ed971c13424aba5fe002377e02e8 [term 1 LEADER]: Becoming Leader. State: Replica: df86ed971c13424aba5fe002377e02e8, State: Running, Role: LEADER
I20260620 11:42:32.946365 8714 consensus_queue.cc:237] T 2308e22bab7d486c95224c056c7e72b4 P df86ed971c13424aba5fe002377e02e8 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df86ed971c13424aba5fe002377e02e8" member_type: VOTER last_known_addr { host: "127.6.234.193" port: 42909 } } peers { permanent_uuid: "08b45869e2ab438dba72a0323d9a56bb" member_type: VOTER last_known_addr { host: "127.6.234.194" port: 37745 } } peers { permanent_uuid: "eab50d9c8b4e4cf6afe28767b40bd8f4" member_type: VOTER last_known_addr { host: "127.6.234.195" port: 33411 } }
I20260620 11:42:32.946656 8722 tablet_replica.cc:439] T 2308e22bab7d486c95224c056c7e72b4 P df86ed971c13424aba5fe002377e02e8: TxnStatusTablet state changed. Reason: New leader df86ed971c13424aba5fe002377e02e8. Latest consensus state: current_term: 1 leader_uuid: "df86ed971c13424aba5fe002377e02e8" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df86ed971c13424aba5fe002377e02e8" member_type: VOTER last_known_addr { host: "127.6.234.193" port: 42909 } } peers { permanent_uuid: "08b45869e2ab438dba72a0323d9a56bb" member_type: VOTER last_known_addr { host: "127.6.234.194" port: 37745 } } peers { permanent_uuid: "eab50d9c8b4e4cf6afe28767b40bd8f4" member_type: VOTER last_known_addr { host: "127.6.234.195" port: 33411 } } }
I20260620 11:42:32.946745 8722 tablet_replica.cc:442] T 2308e22bab7d486c95224c056c7e72b4 P df86ed971c13424aba5fe002377e02e8: This TxnStatusTablet replica's current role is: LEADER
I20260620 11:42:32.946892 8744 txn_status_manager.cc:873] Waiting until node catch up with all replicated operations in previous term...
I20260620 11:42:32.947021 8744 txn_status_manager.cc:929] Loading transaction status metadata into memory...
I20260620 11:42:32.946983 8197 catalog_manager.cc:5495] T 2308e22bab7d486c95224c056c7e72b4 P df86ed971c13424aba5fe002377e02e8 reported cstate change: term changed from 0 to 1, leader changed from <none> to df86ed971c13424aba5fe002377e02e8 (127.6.234.193). New cstate: current_term: 1 leader_uuid: "df86ed971c13424aba5fe002377e02e8" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df86ed971c13424aba5fe002377e02e8" member_type: VOTER last_known_addr { host: "127.6.234.193" port: 42909 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "08b45869e2ab438dba72a0323d9a56bb" member_type: VOTER last_known_addr { host: "127.6.234.194" port: 37745 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "eab50d9c8b4e4cf6afe28767b40bd8f4" member_type: VOTER last_known_addr { host: "127.6.234.195" port: 33411 } health_report { overall_health: UNKNOWN } } }
I20260620 11:42:32.963385 8703 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 11:42:32.960491 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34612 (local address 127.6.234.193:42909)
0620 11:42:32.960538 (+ 47us) server_negotiation.cc:206] Beginning negotiation
0620 11:42:32.960540 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 11:42:32.960608 (+ 68us) server_negotiation.cc:407] Connection header received
0620 11:42:32.960656 (+ 48us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 11:42:32.960657 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 11:42:32.960667 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 11:42:32.960731 (+ 64us) server_negotiation.cc:226] Negotiated authn=TOKEN
0620 11:42:32.960816 (+ 85us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 11:42:32.961576 (+ 760us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 11:42:32.962692 (+ 1116us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 11:42:32.962875 (+ 183us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 11:42:32.962954 (+ 79us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 11:42:32.963144 (+ 190us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0620 11:42:32.963175 (+ 31us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0620 11:42:32.963229 (+ 54us) server_negotiation.cc:1035] Waiting for connection context
0620 11:42:32.963312 (+ 83us) server_negotiation.cc:299] Negotiation successful
0620 11:42:32.963330 (+ 18us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":23}
I20260620 11:42:32.963440 8739 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 11:42:32.960438 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.234.193:42909 (local address 127.0.0.1:34612)
0620 11:42:32.960532 (+ 94us) negotiation.cc:107] Waiting for socket to connect
0620 11:42:32.960544 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0620 11:42:32.960602 (+ 58us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 11:42:32.960734 (+ 132us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 11:42:32.960735 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 11:42:32.960736 (+ 1us) client_negotiation.cc:189] Negotiated authn=TOKEN
0620 11:42:32.960779 (+ 43us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 11:42:32.960781 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 11:42:32.961676 (+ 895us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 11:42:32.961677 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 11:42:32.962610 (+ 933us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 11:42:32.962613 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 11:42:32.962958 (+ 345us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 11:42:32.962959 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 11:42:32.963026 (+ 67us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 11:42:32.963044 (+ 18us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0620 11:42:32.963246 (+ 202us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0620 11:42:32.963254 (+ 8us) client_negotiation.cc:769] Sending connection context
0620 11:42:32.963310 (+ 56us) client_negotiation.cc:240] Negotiation successful
0620 11:42:32.963360 (+ 50us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":53}
I20260620 11:42:32.964370 8495 raft_consensus.cc:1261] T 2308e22bab7d486c95224c056c7e72b4 P 08b45869e2ab438dba72a0323d9a56bb [term 1 FOLLOWER]: Refusing update from remote peer df86ed971c13424aba5fe002377e02e8: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 11:42:32.964365 8635 raft_consensus.cc:1261] T 2308e22bab7d486c95224c056c7e72b4 P eab50d9c8b4e4cf6afe28767b40bd8f4 [term 1 FOLLOWER]: Refusing update from remote peer df86ed971c13424aba5fe002377e02e8: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 11:42:32.964615 8722 consensus_queue.cc:1035] T 2308e22bab7d486c95224c056c7e72b4 P df86ed971c13424aba5fe002377e02e8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "08b45869e2ab438dba72a0323d9a56bb" member_type: VOTER last_known_addr { host: "127.6.234.194" port: 37745 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 11:42:32.964742 8722 consensus_queue.cc:1035] T 2308e22bab7d486c95224c056c7e72b4 P df86ed971c13424aba5fe002377e02e8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "eab50d9c8b4e4cf6afe28767b40bd8f4" member_type: VOTER last_known_addr { host: "127.6.234.195" port: 33411 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 11:42:32.965190 8710 tablet_replica.cc:439] T 2308e22bab7d486c95224c056c7e72b4 P eab50d9c8b4e4cf6afe28767b40bd8f4: TxnStatusTablet state changed. Reason: New leader df86ed971c13424aba5fe002377e02e8. Latest consensus state: current_term: 1 leader_uuid: "df86ed971c13424aba5fe002377e02e8" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df86ed971c13424aba5fe002377e02e8" member_type: VOTER last_known_addr { host: "127.6.234.193" port: 42909 } } peers { permanent_uuid: "08b45869e2ab438dba72a0323d9a56bb" member_type: VOTER last_known_addr { host: "127.6.234.194" port: 37745 } } peers { permanent_uuid: "eab50d9c8b4e4cf6afe28767b40bd8f4" member_type: VOTER last_known_addr { host: "127.6.234.195" port: 33411 } } }
I20260620 11:42:32.965276 8710 tablet_replica.cc:442] T 2308e22bab7d486c95224c056c7e72b4 P eab50d9c8b4e4cf6afe28767b40bd8f4: This TxnStatusTablet replica's current role is: FOLLOWER
I20260620 11:42:32.965487 8712 tablet_replica.cc:439] T 2308e22bab7d486c95224c056c7e72b4 P 08b45869e2ab438dba72a0323d9a56bb: TxnStatusTablet state changed. Reason: New leader df86ed971c13424aba5fe002377e02e8. Latest consensus state: current_term: 1 leader_uuid: "df86ed971c13424aba5fe002377e02e8" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df86ed971c13424aba5fe002377e02e8" member_type: VOTER last_known_addr { host: "127.6.234.193" port: 42909 } } peers { permanent_uuid: "08b45869e2ab438dba72a0323d9a56bb" member_type: VOTER last_known_addr { host: "127.6.234.194" port: 37745 } } peers { permanent_uuid: "eab50d9c8b4e4cf6afe28767b40bd8f4" member_type: VOTER last_known_addr { host: "127.6.234.195" port: 33411 } } }
I20260620 11:42:32.965590 8712 tablet_replica.cc:442] T 2308e22bab7d486c95224c056c7e72b4 P 08b45869e2ab438dba72a0323d9a56bb: This TxnStatusTablet replica's current role is: FOLLOWER
I20260620 11:42:32.965639 8714 tablet_replica.cc:439] T 2308e22bab7d486c95224c056c7e72b4 P df86ed971c13424aba5fe002377e02e8: TxnStatusTablet state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "df86ed971c13424aba5fe002377e02e8" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df86ed971c13424aba5fe002377e02e8" member_type: VOTER last_known_addr { host: "127.6.234.193" port: 42909 } } peers { permanent_uuid: "08b45869e2ab438dba72a0323d9a56bb" member_type: VOTER last_known_addr { host: "127.6.234.194" port: 37745 } } peers { permanent_uuid: "eab50d9c8b4e4cf6afe28767b40bd8f4" member_type: VOTER last_known_addr { host: "127.6.234.195" port: 33411 } } }
I20260620 11:42:32.965729 8714 tablet_replica.cc:442] T 2308e22bab7d486c95224c056c7e72b4 P df86ed971c13424aba5fe002377e02e8: This TxnStatusTablet replica's current role is: LEADER
I20260620 11:42:32.965791 8717 mvcc.cc:205] Tried to move back new op lower bound from 7298890764140281856 to 7298890764068605952. Current Snapshot: MvccSnapshot[applied={T|T < 7298890764140281856 or (T in {7298890764140281856})}]
I20260620 11:42:32.966228 8712 tablet_replica.cc:439] T 2308e22bab7d486c95224c056c7e72b4 P 08b45869e2ab438dba72a0323d9a56bb: TxnStatusTablet state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "df86ed971c13424aba5fe002377e02e8" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df86ed971c13424aba5fe002377e02e8" member_type: VOTER last_known_addr { host: "127.6.234.193" port: 42909 } } peers { permanent_uuid: "08b45869e2ab438dba72a0323d9a56bb" member_type: VOTER last_known_addr { host: "127.6.234.194" port: 37745 } } peers { permanent_uuid: "eab50d9c8b4e4cf6afe28767b40bd8f4" member_type: VOTER last_known_addr { host: "127.6.234.195" port: 33411 } } }
I20260620 11:42:32.966310 8712 tablet_replica.cc:442] T 2308e22bab7d486c95224c056c7e72b4 P 08b45869e2ab438dba72a0323d9a56bb: This TxnStatusTablet replica's current role is: FOLLOWER
I20260620 11:42:32.966583 8722 tablet_replica.cc:439] T 2308e22bab7d486c95224c056c7e72b4 P df86ed971c13424aba5fe002377e02e8: TxnStatusTablet state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "df86ed971c13424aba5fe002377e02e8" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df86ed971c13424aba5fe002377e02e8" member_type: VOTER last_known_addr { host: "127.6.234.193" port: 42909 } } peers { permanent_uuid: "08b45869e2ab438dba72a0323d9a56bb" member_type: VOTER last_known_addr { host: "127.6.234.194" port: 37745 } } peers { permanent_uuid: "eab50d9c8b4e4cf6afe28767b40bd8f4" member_type: VOTER last_known_addr { host: "127.6.234.195" port: 33411 } } }
I20260620 11:42:32.966668 8722 tablet_replica.cc:442] T 2308e22bab7d486c95224c056c7e72b4 P df86ed971c13424aba5fe002377e02e8: This TxnStatusTablet replica's current role is: LEADER
I20260620 11:42:32.967044 8710 tablet_replica.cc:439] T 2308e22bab7d486c95224c056c7e72b4 P eab50d9c8b4e4cf6afe28767b40bd8f4: TxnStatusTablet state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "df86ed971c13424aba5fe002377e02e8" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "df86ed971c13424aba5fe002377e02e8" member_type: VOTER last_known_addr { host: "127.6.234.193" port: 42909 } } peers { permanent_uuid: "08b45869e2ab438dba72a0323d9a56bb" member_type: VOTER last_known_addr { host: "127.6.234.194" port: 37745 } } peers { permanent_uuid: "eab50d9c8b4e4cf6afe28767b40bd8f4" member_type: VOTER last_known_addr { host: "127.6.234.195" port: 33411 } } }
I20260620 11:42:32.967125 8710 tablet_replica.cc:442] T 2308e22bab7d486c95224c056c7e72b4 P eab50d9c8b4e4cf6afe28767b40bd8f4: This TxnStatusTablet replica's current role is: FOLLOWER
I20260620 11:42:32.971658 8703 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 11:42:32.969099 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.6.234.193:55533 (local address 127.6.234.193:42909)
0620 11:42:32.969138 (+ 39us) server_negotiation.cc:206] Beginning negotiation
0620 11:42:32.969140 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 11:42:32.969316 (+ 176us) server_negotiation.cc:407] Connection header received
0620 11:42:32.969379 (+ 63us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 11:42:32.969380 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 11:42:32.969387 (+ 7us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 11:42:32.969431 (+ 44us) server_negotiation.cc:226] Negotiated authn=TOKEN
0620 11:42:32.969533 (+ 102us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 11:42:32.970033 (+ 500us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 11:42:32.971096 (+ 1063us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 11:42:32.971264 (+ 168us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 11:42:32.971311 (+ 47us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 11:42:32.971476 (+ 165us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0620 11:42:32.971497 (+ 21us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0620 11:42:32.971533 (+ 36us) server_negotiation.cc:1035] Waiting for connection context
0620 11:42:32.971585 (+ 52us) server_negotiation.cc:299] Negotiation successful
0620 11:42:32.971598 (+ 13us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":18}
I20260620 11:42:32.971740 8751 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 11:42:32.969061 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.234.193:42909 (local address 127.6.234.193:55533)
0620 11:42:32.969250 (+ 189us) negotiation.cc:107] Waiting for socket to connect
0620 11:42:32.969262 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0620 11:42:32.969323 (+ 61us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 11:42:32.969434 (+ 111us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 11:42:32.969435 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 11:42:32.969437 (+ 2us) client_negotiation.cc:189] Negotiated authn=TOKEN
0620 11:42:32.969479 (+ 42us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 11:42:32.969480 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 11:42:32.970111 (+ 631us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 11:42:32.970112 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 11:42:32.971014 (+ 902us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 11:42:32.971016 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 11:42:32.971331 (+ 315us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 11:42:32.971332 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 11:42:32.971388 (+ 56us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 11:42:32.971403 (+ 15us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0620 11:42:32.971534 (+ 131us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0620 11:42:32.971536 (+ 2us) client_negotiation.cc:769] Sending connection context
0620 11:42:32.971583 (+ 47us) client_negotiation.cc:240] Negotiation successful
0620 11:42:32.971625 (+ 42us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":149,"thread_start_us":98,"threads_started":1}
I20260620 11:42:32.973869 8635 raft_consensus.cc:1261] T b2e3e3c341854e3ea73763f4a8326c8d P eab50d9c8b4e4cf6afe28767b40bd8f4 [term 1 FOLLOWER]: Refusing update from remote peer df86ed971c13424aba5fe002377e02e8: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 11:42:32.973897 8495 raft_consensus.cc:1261] T b2e3e3c341854e3ea73763f4a8326c8d P 08b45869e2ab438dba72a0323d9a56bb [term 1 FOLLOWER]: Refusing update from remote peer df86ed971c13424aba5fe002377e02e8: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 11:42:32.974063 8722 consensus_queue.cc:1035] T b2e3e3c341854e3ea73763f4a8326c8d P df86ed971c13424aba5fe002377e02e8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "eab50d9c8b4e4cf6afe28767b40bd8f4" member_type: VOTER last_known_addr { host: "127.6.234.195" port: 33411 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 11:42:32.974260 8722 consensus_queue.cc:1035] T b2e3e3c341854e3ea73763f4a8326c8d P df86ed971c13424aba5fe002377e02e8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "08b45869e2ab438dba72a0323d9a56bb" member_type: VOTER last_known_addr { host: "127.6.234.194" port: 37745 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 11:42:32.985055 8197 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:46864:
table { table_name: "test-table" } modify_external_catalogs: true
I20260620 11:42:32.985167 8197 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:46864:
table { table_name: "test-table" } modify_external_catalogs: true
I20260620 11:42:32.985783 8197 catalog_manager.cc:5782] T 00000000000000000000000000000000 P a56f5849e11d410fa2127cc9991f437c: Sending DeleteTablet for 3 replicas of tablet b2e3e3c341854e3ea73763f4a8326c8d
I20260620 11:42:32.986012 8335 tablet_service.cc:1514] Processing DeleteTablet for tablet b2e3e3c341854e3ea73763f4a8326c8d with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 11:42:32 UTC) from {username='kudu', principal='kudu/127.6.234.254@KRBTEST.COM'} at 127.0.0.1:57570
I20260620 11:42:32.986025 8475 tablet_service.cc:1514] Processing DeleteTablet for tablet b2e3e3c341854e3ea73763f4a8326c8d with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 11:42:32 UTC) from {username='kudu', principal='kudu/127.6.234.254@KRBTEST.COM'} at 127.0.0.1:52026
I20260620 11:42:32.986112 8729 tablet_replica.cc:330] T b2e3e3c341854e3ea73763f4a8326c8d P 08b45869e2ab438dba72a0323d9a56bb: stopping tablet replica
I20260620 11:42:32.986110 8727 tablet_replica.cc:330] T b2e3e3c341854e3ea73763f4a8326c8d P df86ed971c13424aba5fe002377e02e8: stopping tablet replica
I20260620 11:42:32.986155 8729 raft_consensus.cc:2229] T b2e3e3c341854e3ea73763f4a8326c8d P 08b45869e2ab438dba72a0323d9a56bb [term 1 FOLLOWER]: Raft consensus shutting down.
I20260620 11:42:32.986157 8727 raft_consensus.cc:2229] T b2e3e3c341854e3ea73763f4a8326c8d P df86ed971c13424aba5fe002377e02e8 [term 1 LEADER]: Raft consensus shutting down.
I20260620 11:42:32.986189 8729 raft_consensus.cc:2258] T b2e3e3c341854e3ea73763f4a8326c8d P 08b45869e2ab438dba72a0323d9a56bb [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 11:42:32.986212 8727 raft_consensus.cc:2258] T b2e3e3c341854e3ea73763f4a8326c8d P df86ed971c13424aba5fe002377e02e8 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 11:42:32.986317 8615 tablet_service.cc:1514] Processing DeleteTablet for tablet b2e3e3c341854e3ea73763f4a8326c8d with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 11:42:32 UTC) from {username='kudu', principal='kudu/127.6.234.254@KRBTEST.COM'} at 127.0.0.1:49788
I20260620 11:42:32.986416 8728 tablet_replica.cc:330] T b2e3e3c341854e3ea73763f4a8326c8d P eab50d9c8b4e4cf6afe28767b40bd8f4: stopping tablet replica
I20260620 11:42:32.986481 8728 raft_consensus.cc:2229] T b2e3e3c341854e3ea73763f4a8326c8d P eab50d9c8b4e4cf6afe28767b40bd8f4 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260620 11:42:32.986523 8728 raft_consensus.cc:2258] T b2e3e3c341854e3ea73763f4a8326c8d P eab50d9c8b4e4cf6afe28767b40bd8f4 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 11:42:32.986511 8729 ts_tablet_manager.cc:1905] T b2e3e3c341854e3ea73763f4a8326c8d P 08b45869e2ab438dba72a0323d9a56bb: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 11:42:32.986838 8728 ts_tablet_manager.cc:1905] T b2e3e3c341854e3ea73763f4a8326c8d P eab50d9c8b4e4cf6afe28767b40bd8f4: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 11:42:32.987991 8727 ts_tablet_manager.cc:1905] T b2e3e3c341854e3ea73763f4a8326c8d P df86ed971c13424aba5fe002377e02e8: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 11:42:32.988159 8729 ts_tablet_manager.cc:1918] T b2e3e3c341854e3ea73763f4a8326c8d P 08b45869e2ab438dba72a0323d9a56bb: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260620 11:42:32.988220 8729 log.cc:1196] T b2e3e3c341854e3ea73763f4a8326c8d P 08b45869e2ab438dba72a0323d9a56bb: Deleting WAL directory at /tmp/dist-test-task1JRg7N/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781955746769051-7083-0/minicluster-data/ts-1/wal/wals/b2e3e3c341854e3ea73763f4a8326c8d
I20260620 11:42:32.988425 8729 ts_tablet_manager.cc:1939] T b2e3e3c341854e3ea73763f4a8326c8d P 08b45869e2ab438dba72a0323d9a56bb: Deleting consensus metadata
I20260620 11:42:32.988709 8185 catalog_manager.cc:4841] TS 08b45869e2ab438dba72a0323d9a56bb (127.6.234.194:37745): tablet b2e3e3c341854e3ea73763f4a8326c8d (table test-table [id=c1a0da12220e4e86bee6aa9a8aef4fe3]) successfully deleted
I20260620 11:42:32.989406 8727 ts_tablet_manager.cc:1918] T b2e3e3c341854e3ea73763f4a8326c8d P df86ed971c13424aba5fe002377e02e8: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260620 11:42:32.989454 8727 log.cc:1196] T b2e3e3c341854e3ea73763f4a8326c8d P df86ed971c13424aba5fe002377e02e8: Deleting WAL directory at /tmp/dist-test-task1JRg7N/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781955746769051-7083-0/minicluster-data/ts-0/wal/wals/b2e3e3c341854e3ea73763f4a8326c8d
I20260620 11:42:32.989674 8727 ts_tablet_manager.cc:1939] T b2e3e3c341854e3ea73763f4a8326c8d P df86ed971c13424aba5fe002377e02e8: Deleting consensus metadata
I20260620 11:42:32.989997 8728 ts_tablet_manager.cc:1918] T b2e3e3c341854e3ea73763f4a8326c8d P eab50d9c8b4e4cf6afe28767b40bd8f4: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260620 11:42:32.990020 8182 catalog_manager.cc:4841] TS df86ed971c13424aba5fe002377e02e8 (127.6.234.193:42909): tablet b2e3e3c341854e3ea73763f4a8326c8d (table test-table [id=c1a0da12220e4e86bee6aa9a8aef4fe3]) successfully deleted
I20260620 11:42:32.990062 8728 log.cc:1196] T b2e3e3c341854e3ea73763f4a8326c8d P eab50d9c8b4e4cf6afe28767b40bd8f4: Deleting WAL directory at /tmp/dist-test-task1JRg7N/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781955746769051-7083-0/minicluster-data/ts-2/wal/wals/b2e3e3c341854e3ea73763f4a8326c8d
I20260620 11:42:32.990288 8728 ts_tablet_manager.cc:1939] T b2e3e3c341854e3ea73763f4a8326c8d P eab50d9c8b4e4cf6afe28767b40bd8f4: Deleting consensus metadata
I20260620 11:42:32.990597 8185 catalog_manager.cc:4841] TS eab50d9c8b4e4cf6afe28767b40bd8f4 (127.6.234.195:33411): tablet b2e3e3c341854e3ea73763f4a8326c8d (table test-table [id=c1a0da12220e4e86bee6aa9a8aef4fe3]) successfully deleted
Jun 20 11:42:32 dist-test-slave-cjc6 krb5kdc[8153](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781955752, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for kudu/127.6.234.193@KRBTEST.COM
I20260620 11:42:32.997249 8703 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 11:42:32.987119 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34626 (local address 127.6.234.193:42909)
0620 11:42:32.987158 (+ 39us) server_negotiation.cc:206] Beginning negotiation
0620 11:42:32.987160 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 11:42:32.989886 (+ 2726us) server_negotiation.cc:407] Connection header received
0620 11:42:32.989952 (+ 66us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 11:42:32.989953 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 11:42:32.989964 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 11:42:32.989994 (+ 30us) server_negotiation.cc:226] Negotiated authn=SASL
0620 11:42:32.990788 (+ 794us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 11:42:32.991259 (+ 471us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 11:42:32.991955 (+ 696us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 11:42:32.992167 (+ 212us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 11:42:32.992242 (+ 75us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 11:42:32.994524 (+ 2282us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 11:42:32.994533 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 11:42:32.994533 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 11:42:32.994540 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0620 11:42:32.996081 (+ 1541us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 11:42:32.996426 (+ 345us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 11:42:32.996430 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 11:42:32.996431 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 11:42:32.996499 (+ 68us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 11:42:32.996715 (+ 216us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 11:42:32.996716 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 11:42:32.996717 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 11:42:32.996856 (+ 139us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 11:42:32.997022 (+ 166us) server_negotiation.cc:1035] Waiting for connection context
0620 11:42:32.997067 (+ 45us) server_negotiation.cc:299] Negotiation successful
0620 11:42:32.997152 (+ 85us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":18}
W20260620 11:42:32.997489 8388 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.SetFlag from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:34626
I20260620 11:42:33.004521 8248 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 11:42:32.998455 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41788 (local address 127.6.234.254:43185)
0620 11:42:32.998515 (+ 60us) server_negotiation.cc:206] Beginning negotiation
0620 11:42:32.998517 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 11:42:32.998566 (+ 49us) server_negotiation.cc:407] Connection header received
0620 11:42:32.998615 (+ 49us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 11:42:32.998616 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 11:42:32.998626 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 11:42:32.998692 (+ 66us) server_negotiation.cc:226] Negotiated authn=SASL
0620 11:42:32.999101 (+ 409us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 11:42:32.999850 (+ 749us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 11:42:33.000563 (+ 713us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 11:42:33.000866 (+ 303us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 11:42:33.000946 (+ 80us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 11:42:33.002082 (+ 1136us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 11:42:33.002084 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 11:42:33.002084 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 11:42:33.002091 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0620 11:42:33.003400 (+ 1309us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 11:42:33.003750 (+ 350us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 11:42:33.003751 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 11:42:33.003752 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 11:42:33.003806 (+ 54us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 11:42:33.004013 (+ 207us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 11:42:33.004014 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 11:42:33.004015 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 11:42:33.004140 (+ 125us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 11:42:33.004246 (+ 106us) server_negotiation.cc:1035] Waiting for connection context
0620 11:42:33.004350 (+ 104us) server_negotiation.cc:299] Negotiation successful
0620 11:42:33.004423 (+ 73us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":36}
W20260620 11:42:33.004815 8197 server_base.cc:1116] Unauthorized access attempt to method kudu.master.MasterService.TSHeartbeat from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:41788
I20260620 11:42:33.005293 7083 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task1JRg7N/build/release/bin/kudu with pid 8270
I20260620 11:42:33.011137 7083 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task1JRg7N/build/release/bin/kudu with pid 8410
I20260620 11:42:33.015841 7083 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task1JRg7N/build/release/bin/kudu with pid 8550
I20260620 11:42:33.019958 7083 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task1JRg7N/build/release/bin/kudu with pid 8166
2026-06-20T11:42:33Z chronyd exiting
[ OK ] SecurityITest.SmokeTestAsAuthorizedUser (2456 ms)
[ RUN ] SecurityITest.TxnSmokeWithDifferentUserTypes
Loading random data
Initializing database '/tmp/dist-test-task1JRg7N/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781955746769051-7083-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 20 11:42:33 dist-test-slave-cjc6 krb5kdc[8762](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 20 11:42:33 dist-test-slave-cjc6 krb5kdc[8762](info): set up 2 sockets
Jun 20 11:42:33 dist-test-slave-cjc6 krb5kdc[8762](info): commencing operation
krb5kdc: starting...
W20260620 11:42:34.045274 7083 mini_kdc.cc:121] Time spent starting KDC: real 1.005s user 0.004s sys 0.000s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task1JRg7N/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781955746769051-7083-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task1JRg7N/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781955746769051-7083-0/krb5kdc/test-user.keytab.
Jun 20 11:42:34 dist-test-slave-cjc6 krb5kdc[8762](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781955754, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-20T11:42:34Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-20T11:42:34Z Disabled control of system clock
WARNING: no policy specified for kudu/127.6.234.254@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.6.234.254@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.6.234.254 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task1JRg7N/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781955746769051-7083-0/krb5kdc/kudu_127.6.234.254.keytab.
Entry for principal kudu/127.6.234.254 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task1JRg7N/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781955746769051-7083-0/krb5kdc/kudu_127.6.234.254.keytab.
I20260620 11:42:34.156838 7083 external_mini_cluster.cc:1326] Running /tmp/dist-test-task1JRg7N/build/release/bin/kudu
/tmp/dist-test-task1JRg7N/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task1JRg7N/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781955746769051-7083-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task1JRg7N/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781955746769051-7083-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task1JRg7N/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781955746769051-7083-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task1JRg7N/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781955746769051-7083-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.6.234.254:36185
--webserver_interface=127.6.234.254
--webserver_port=0
--keytab_file=/tmp/dist-test-task1JRg7N/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781955746769051-7083-0/krb5kdc/kudu_127.6.234.254.keytab
--principal=kudu/127.6.234.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.6.234.212:37773
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.6.234.254:36185
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--txn_manager_enabled=true with env {KRB5CCNAME=/tmp/dist-test-task1JRg7N/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781955746769051-7083-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task1JRg7N/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781955746769051-7083-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task1JRg7N/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781955746769051-7083-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 11:42:34.236275 8775 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 11:42:34.236502 8775 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 11:42:34.236537 8775 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 11:42:34.237646 8775 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 11:42:34.237730 8775 flags.cc:425] Enabled experimental flag: --txn_manager_enabled=true
W20260620 11:42:34.237754 8775 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 11:42:34.237778 8775 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 11:42:34.237807 8775 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 11:42:34.237834 8775 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 11:42:34.239298 8775 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.6.234.212:37773
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task1JRg7N/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781955746769051-7083-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task1JRg7N/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781955746769051-7083-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.6.234.254:36185
--txn_manager_enabled=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task1JRg7N/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781955746769051-7083-0/krb5kdc/kudu_127.6.234.254.keytab
--principal=kudu/127.6.234.254
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.6.234.254:36185
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task1JRg7N/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781955746769051-7083-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.6.234.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.8775
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task1JRg7N/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781955746769051-7083-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 20 Jun 2026 10:32:37 UTC on ccycloud.quasar-bgyofv.root.comops.site
build id 4457173
I20260620 11:42:34.239566 8775 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 11:42:34.239780 8775 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 11:42:34.244640 8780 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 11:42:34.244624 8781 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 11:42:34.244624 8783 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 11:42:34.244776 8775 server_base.cc:1034] running on GCE node
I20260620 11:42:34.245137 8775 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 11:42:34.245417 8775 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 11:42:34.246584 8775 hybrid_clock.cc:648] HybridClock initialized: now 1781955754246584 us; error 43 us; skew 500 ppm
Jun 20 11:42:34 dist-test-slave-cjc6 krb5kdc[8762](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781955754, etypes {rep=17 tkt=17 ses=17}, kudu/127.6.234.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 11:42:34.248893 8775 init.cc:377] Logged in from keytab as kudu/127.6.234.254@KRBTEST.COM (short username kudu)
I20260620 11:42:34.249320 8775 webserver.cc:460] Webserver started at http://127.6.234.254:33737/ using document root <none> and password file <none>
I20260620 11:42:34.249505 8775 fs_manager.cc:359] Metadata directory not provided
I20260620 11:42:34.249544 8775 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 11:42:34.249629 8775 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 11:42:34.250699 8775 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task1JRg7N/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781955746769051-7083-0/minicluster-data/master-0/data/instance:
uuid: "6e89f37d51dc404c95261d264710ef80"
format_stamp: "Formatted at 2026-06-20 11:42:34 on dist-test-slave-cjc6"
server_key: "491e00704c93cb814c91cb78cc35ec40"
server_key_iv: "617b8a24aae5505ba0a9863504e4296d"
server_key_version: "encryptionkey@0"
I20260620 11:42:34.250991 8775 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task1JRg7N/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781955746769051-7083-0/minicluster-data/master-0/wal/instance:
uuid: "6e89f37d51dc404c95261d264710ef80"
format_stamp: "Formatted at 2026-06-20 11:42:34 on dist-test-slave-cjc6"
server_key: "491e00704c93cb814c91cb78cc35ec40"
server_key_iv: "617b8a24aae5505ba0a9863504e4296d"
server_key_version: "encryptionkey@0"
I20260620 11:42:34.252178 8775 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.002s
I20260620 11:42:34.252810 8790 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 11:42:34.252960 8775 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.001s
I20260620 11:42:34.253042 8775 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task1JRg7N/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781955746769051-7083-0/minicluster-data/master-0/data,/tmp/dist-test-task1JRg7N/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781955746769051-7083-0/minicluster-data/master-0/wal
uuid: "6e89f37d51dc404c95261d264710ef80"
format_stamp: "Formatted at 2026-06-20 11:42:34 on dist-test-slave-cjc6"
server_key: "491e00704c93cb814c91cb78cc35ec40"
server_key_iv: "617b8a24aae5505ba0a9863504e4296d"
server_key_version: "encryptionkey@0"
I20260620 11:42:34.253094 8775 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task1JRg7N/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781955746769051-7083-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task1JRg7N/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781955746769051-7083-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task1JRg7N/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781955746769051-7083-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 11:42:34.276593 8775 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 11:42:34.278548 8775 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 11:42:34.278652 8775 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 11:42:34.282959 8775 rpc_server.cc:307] RPC server started. Bound to: 127.6.234.254:36185
I20260620 11:42:34.283098 8852 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.6.234.254:36185 every 8 connection(s)
I20260620 11:42:34.283330 8775 server_base.cc:1166] Dumped server information to /tmp/dist-test-task1JRg7N/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781955746769051-7083-0/minicluster-data/master-0/data/info.pb
I20260620 11:42:34.284030 8853 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 11:42:34.286235 8853 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 6e89f37d51dc404c95261d264710ef80: Bootstrap starting.
I20260620 11:42:34.286814 8853 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 6e89f37d51dc404c95261d264710ef80: Neither blocks nor log segments found. Creating new log.
I20260620 11:42:34.287096 8853 log.cc:824] T 00000000000000000000000000000000 P 6e89f37d51dc404c95261d264710ef80: Log is configured to *not* fsync() on all Append() calls
I20260620 11:42:34.287767 8853 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 6e89f37d51dc404c95261d264710ef80: No bootstrap required, opened a new log
I20260620 11:42:34.289868 8853 raft_consensus.cc:348] T 00000000000000000000000000000000 P 6e89f37d51dc404c95261d264710ef80 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6e89f37d51dc404c95261d264710ef80" member_type: VOTER last_known_addr { host: "127.6.234.254" port: 36185 } }
I20260620 11:42:34.290027 8853 raft_consensus.cc:374] T 00000000000000000000000000000000 P 6e89f37d51dc404c95261d264710ef80 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 11:42:34.290073 8853 raft_consensus.cc:729] T 00000000000000000000000000000000 P 6e89f37d51dc404c95261d264710ef80 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6e89f37d51dc404c95261d264710ef80, State: Initialized, Role: FOLLOWER
I20260620 11:42:34.290213 8853 consensus_queue.cc:260] T 00000000000000000000000000000000 P 6e89f37d51dc404c95261d264710ef80 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6e89f37d51dc404c95261d264710ef80" member_type: VOTER last_known_addr { host: "127.6.234.254" port: 36185 } }
I20260620 11:42:34.290290 8853 raft_consensus.cc:388] T 00000000000000000000000000000000 P 6e89f37d51dc404c95261d264710ef80 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260620 11:42:34.290333 8853 raft_consensus.cc:482] T 00000000000000000000000000000000 P 6e89f37d51dc404c95261d264710ef80 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260620 11:42:34.290392 8853 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 6e89f37d51dc404c95261d264710ef80 [term 0 FOLLOWER]: Advancing to term 1
I20260620 11:42:34.290892 8853 raft_consensus.cc:504] T 00000000000000000000000000000000 P 6e89f37d51dc404c95261d264710ef80 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6e89f37d51dc404c95261d264710ef80" member_type: VOTER last_known_addr { host: "127.6.234.254" port: 36185 } }
I20260620 11:42:34.291005 8853 leader_election.cc:302] T 00000000000000000000000000000000 P 6e89f37d51dc404c95261d264710ef80 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 6e89f37d51dc404c95261d264710ef80; no voters:
I20260620 11:42:34.291172 7083 external_mini_cluster.cc:1388] Started /tmp/dist-test-task1JRg7N/build/release/bin/kudu as pid 8775
I20260620 11:42:34.291231 8853 leader_election.cc:288] T 00000000000000000000000000000000 P 6e89f37d51dc404c95261d264710ef80 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260620 11:42:34.291265 8856 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 6e89f37d51dc404c95261d264710ef80 [term 1 FOLLOWER]: Leader election won for term 1
I20260620 11:42:34.291288 7083 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task1JRg7N/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781955746769051-7083-0/minicluster-data/master-0/wal/instance
I20260620 11:42:34.291388 8856 raft_consensus.cc:686] T 00000000000000000000000000000000 P 6e89f37d51dc404c95261d264710ef80 [term 1 LEADER]: Becoming Leader. State: Replica: 6e89f37d51dc404c95261d264710ef80, State: Running, Role: LEADER
I20260620 11:42:34.291461 7083 external_mini_cluster.cc:1428] Setting key 63342a5a66b9e1ab66bbe152e61fc66a
I20260620 11:42:34.291487 8853 sys_catalog.cc:564] T 00000000000000000000000000000000 P 6e89f37d51dc404c95261d264710ef80 [sys.catalog]: configured and running, proceeding with master startup.
I20260620 11:42:34.291468 8856 consensus_queue.cc:237] T 00000000000000000000000000000000 P 6e89f37d51dc404c95261d264710ef80 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6e89f37d51dc404c95261d264710ef80" member_type: VOTER last_known_addr { host: "127.6.234.254" port: 36185 } }
I20260620 11:42:34.292258 8857 sys_catalog.cc:455] T 00000000000000000000000000000000 P 6e89f37d51dc404c95261d264710ef80 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 6e89f37d51dc404c95261d264710ef80. Latest consensus state: current_term: 1 leader_uuid: "6e89f37d51dc404c95261d264710ef80" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6e89f37d51dc404c95261d264710ef80" member_type: VOTER last_known_addr { host: "127.6.234.254" port: 36185 } } }
I20260620 11:42:34.292330 8857 sys_catalog.cc:458] T 00000000000000000000000000000000 P 6e89f37d51dc404c95261d264710ef80 [sys.catalog]: This master's current role is: LEADER
I20260620 11:42:34.292232 8858 sys_catalog.cc:455] T 00000000000000000000000000000000 P 6e89f37d51dc404c95261d264710ef80 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "6e89f37d51dc404c95261d264710ef80" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6e89f37d51dc404c95261d264710ef80" member_type: VOTER last_known_addr { host: "127.6.234.254" port: 36185 } } }
I20260620 11:42:34.292433 8858 sys_catalog.cc:458] T 00000000000000000000000000000000 P 6e89f37d51dc404c95261d264710ef80 [sys.catalog]: This master's current role is: LEADER
I20260620 11:42:34.292781 8867 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260620 11:42:34.293259 8867 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260620 11:42:34.296049 8867 catalog_manager.cc:1347] Generated new cluster ID: 60e627e218f8410f8d64bfe7f760ea4a
I20260620 11:42:34.296105 8867 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
Jun 20 11:42:34 dist-test-slave-cjc6 krb5kdc[8762](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781955754, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.6.234.254@KRBTEST.COM
I20260620 11:42:34.302206 8863 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 11:42:34.291999 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47672 (local address 127.6.234.254:36185)
0620 11:42:34.292312 (+ 313us) server_negotiation.cc:206] Beginning negotiation
0620 11:42:34.292543 (+ 231us) server_negotiation.cc:399] Waiting for connection header
0620 11:42:34.292581 (+ 38us) server_negotiation.cc:407] Connection header received
0620 11:42:34.292880 (+ 299us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 11:42:34.292887 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 11:42:34.292942 (+ 55us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 11:42:34.293091 (+ 149us) server_negotiation.cc:226] Negotiated authn=SASL
0620 11:42:34.293715 (+ 624us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 11:42:34.294904 (+ 1189us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 11:42:34.295980 (+ 1076us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 11:42:34.296445 (+ 465us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 11:42:34.296565 (+ 120us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 11:42:34.298961 (+ 2396us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 11:42:34.298965 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 11:42:34.298965 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 11:42:34.298977 (+ 12us) server_negotiation.cc:922] calling sasl_server_start()
0620 11:42:34.300636 (+ 1659us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 11:42:34.301016 (+ 380us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 11:42:34.301019 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 11:42:34.301020 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 11:42:34.301119 (+ 99us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 11:42:34.301329 (+ 210us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 11:42:34.301330 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 11:42:34.301330 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 11:42:34.301556 (+ 226us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 11:42:34.301748 (+ 192us) server_negotiation.cc:1035] Waiting for connection context
0620 11:42:34.301863 (+ 115us) server_negotiation.cc:299] Negotiation successful
0620 11:42:34.302011 (+ 148us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":131,"thread_start_us":78,"threads_started":1}
I20260620 11:42:34.311764 8867 catalog_manager.cc:1370] Generated new certificate authority record
I20260620 11:42:34.312422 8867 catalog_manager.cc:1504] Loading token signing keys...
I20260620 11:42:34.319598 8867 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 6e89f37d51dc404c95261d264710ef80: Generated new TSK 0
I20260620 11:42:34.319756 8867 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.6.234.193@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.6.234.193@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.6.234.193 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task1JRg7N/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781955746769051-7083-0/krb5kdc/kudu_127.6.234.193.keytab.
Entry for principal kudu/127.6.234.193 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task1JRg7N/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781955746769051-7083-0/krb5kdc/kudu_127.6.234.193.keytab.
I20260620 11:42:34.350315 7083 external_mini_cluster.cc:1326] Running /tmp/dist-test-task1JRg7N/build/release/bin/kudu
/tmp/dist-test-task1JRg7N/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task1JRg7N/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781955746769051-7083-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-task1JRg7N/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781955746769051-7083-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task1JRg7N/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781955746769051-7083-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-task1JRg7N/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781955746769051-7083-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.6.234.193:0
--local_ip_for_outbound_sockets=127.6.234.193
--webserver_interface=127.6.234.193
--webserver_port=0
--tserver_master_addrs=127.6.234.254:36185
--keytab_file=/tmp/dist-test-task1JRg7N/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781955746769051-7083-0/krb5kdc/kudu_127.6.234.193.keytab
--principal=kudu/127.6.234.193
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.6.234.212:37773
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true
--txn_keepalive_interval_ms=500 with env {KRB5CCNAME=/tmp/dist-test-task1JRg7N/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781955746769051-7083-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task1JRg7N/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781955746769051-7083-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task1JRg7N/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781955746769051-7083-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 11:42:34.429536 8879 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 11:42:34.429742 8879 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 11:42:34.429767 8879 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 11:42:34.430912 8879 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 11:42:34.430953 8879 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 11:42:34.430974 8879 flags.cc:425] Enabled experimental flag: --txn_keepalive_interval_ms=500
W20260620 11:42:34.430986 8879 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260620 11:42:34.431002 8879 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.6.234.193
I20260620 11:42:34.432406 8879 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.6.234.212:37773
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task1JRg7N/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781955746769051-7083-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-task1JRg7N/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781955746769051-7083-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task1JRg7N/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781955746769051-7083-0/krb5kdc/kudu_127.6.234.193.keytab
--principal=kudu/127.6.234.193
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.6.234.193:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task1JRg7N/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781955746769051-7083-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.6.234.193
--webserver_port=0
--txn_keepalive_interval_ms=500
--enable_txn_system_client_init=true
--tserver_master_addrs=127.6.234.254:36185
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.8879
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.6.234.193
--log_dir=/tmp/dist-test-task1JRg7N/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781955746769051-7083-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 20 Jun 2026 10:32:37 UTC on ccycloud.quasar-bgyofv.root.comops.site
build id 4457173
I20260620 11:42:34.432646 8879 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 11:42:34.432865 8879 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 11:42:34.437774 8885 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 11:42:34.437772 8884 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 11:42:34.437837 8887 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 11:42:34.437947 8879 server_base.cc:1034] running on GCE node
I20260620 11:42:34.438138 8879 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 11:42:34.438321 8879 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 11:42:34.439456 8879 hybrid_clock.cc:648] HybridClock initialized: now 1781955754439442 us; error 29 us; skew 500 ppm
Jun 20 11:42:34 dist-test-slave-cjc6 krb5kdc[8762](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781955754, etypes {rep=17 tkt=17 ses=17}, kudu/127.6.234.193@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 11:42:34.441861 8879 init.cc:377] Logged in from keytab as kudu/127.6.234.193@KRBTEST.COM (short username kudu)
I20260620 11:42:34.442325 8879 webserver.cc:460] Webserver started at http://127.6.234.193:39477/ using document root <none> and password file <none>
I20260620 11:42:34.442564 8879 fs_manager.cc:359] Metadata directory not provided
I20260620 11:42:34.442632 8879 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 11:42:34.442785 8879 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 11:42:34.443790 8879 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task1JRg7N/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781955746769051-7083-0/minicluster-data/ts-0/data/instance:
uuid: "47814c4c6447409dbab77568f6434cbd"
format_stamp: "Formatted at 2026-06-20 11:42:34 on dist-test-slave-cjc6"
server_key: "5d5fa32ecf9f6b2dc23862c1e196da45"
server_key_iv: "90bdc69a85121de9ee4328f80788a577"
server_key_version: "encryptionkey@0"
I20260620 11:42:34.444168 8879 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task1JRg7N/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781955746769051-7083-0/minicluster-data/ts-0/wal/instance:
uuid: "47814c4c6447409dbab77568f6434cbd"
format_stamp: "Formatted at 2026-06-20 11:42:34 on dist-test-slave-cjc6"
server_key: "5d5fa32ecf9f6b2dc23862c1e196da45"
server_key_iv: "90bdc69a85121de9ee4328f80788a577"
server_key_version: "encryptionkey@0"
I20260620 11:42:34.445562 8879 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260620 11:42:34.446350 8894 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 11:42:34.446597 8879 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260620 11:42:34.446703 8879 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task1JRg7N/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781955746769051-7083-0/minicluster-data/ts-0/data,/tmp/dist-test-task1JRg7N/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781955746769051-7083-0/minicluster-data/ts-0/wal
uuid: "47814c4c6447409dbab77568f6434cbd"
format_stamp: "Formatted at 2026-06-20 11:42:34 on dist-test-slave-cjc6"
server_key: "5d5fa32ecf9f6b2dc23862c1e196da45"
server_key_iv: "90bdc69a85121de9ee4328f80788a577"
server_key_version: "encryptionkey@0"
I20260620 11:42:34.446784 8879 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task1JRg7N/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781955746769051-7083-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-task1JRg7N/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781955746769051-7083-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-task1JRg7N/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781955746769051-7083-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 11:42:34.459096 8879 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 11:42:34.460845 8879 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 11:42:34.460950 8879 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 11:42:34.461349 8879 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 11:42:34.461380 8879 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 11:42:34.461400 8879 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 11:42:34.461444 8879 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 11:42:34.468303 8879 rpc_server.cc:307] RPC server started. Bound to: 127.6.234.193:40543
I20260620 11:42:34.468364 9009 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.6.234.193:40543 every 8 connection(s)
Jun 20 11:42:34 dist-test-slave-cjc6 krb5kdc[8762](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781955754, etypes {rep=17 tkt=17 ses=17}, kudu/127.6.234.193@KRBTEST.COM for kudu/127.6.234.254@KRBTEST.COM
I20260620 11:42:34.468654 8879 server_base.cc:1166] Dumped server information to /tmp/dist-test-task1JRg7N/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781955746769051-7083-0/minicluster-data/ts-0/data/info.pb
I20260620 11:42:34.471635 8863 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 11:42:34.462004 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.6.234.193:58833 (local address 127.6.234.254:36185)
0620 11:42:34.462052 (+ 48us) server_negotiation.cc:206] Beginning negotiation
0620 11:42:34.462054 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 11:42:34.462313 (+ 259us) server_negotiation.cc:407] Connection header received
0620 11:42:34.462837 (+ 524us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 11:42:34.462839 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 11:42:34.462851 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 11:42:34.462940 (+ 89us) server_negotiation.cc:226] Negotiated authn=SASL
0620 11:42:34.463797 (+ 857us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 11:42:34.464658 (+ 861us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 11:42:34.466068 (+ 1410us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 11:42:34.466400 (+ 332us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 11:42:34.466487 (+ 87us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 11:42:34.469225 (+ 2738us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 11:42:34.469227 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 11:42:34.469227 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 11:42:34.469233 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0620 11:42:34.470468 (+ 1235us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 11:42:34.470819 (+ 351us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 11:42:34.470820 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 11:42:34.470820 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 11:42:34.470868 (+ 48us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 11:42:34.471089 (+ 221us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 11:42:34.471090 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 11:42:34.471090 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 11:42:34.471185 (+ 95us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 11:42:34.471235 (+ 50us) server_negotiation.cc:1035] Waiting for connection context
0620 11:42:34.471515 (+ 280us) server_negotiation.cc:299] Negotiation successful
0620 11:42:34.471575 (+ 60us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":21}
I20260620 11:42:34.471859 8903 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 11:42:34.462018 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.234.254:36185 (local address 127.6.234.193:58833)
0620 11:42:34.462190 (+ 172us) negotiation.cc:107] Waiting for socket to connect
0620 11:42:34.462222 (+ 32us) client_negotiation.cc:174] Beginning negotiation
0620 11:42:34.462736 (+ 514us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 11:42:34.463011 (+ 275us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 11:42:34.463015 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 11:42:34.463396 (+ 381us) client_negotiation.cc:189] Negotiated authn=SASL
0620 11:42:34.463679 (+ 283us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 11:42:34.463685 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 11:42:34.464761 (+ 1076us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 11:42:34.464762 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 11:42:34.465997 (+ 1235us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 11:42:34.466000 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 11:42:34.466508 (+ 508us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 11:42:34.466509 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 11:42:34.466609 (+ 100us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 11:42:34.467227 (+ 618us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 11:42:34.467237 (+ 10us) client_negotiation.cc:656] calling sasl_client_start()
0620 11:42:34.469130 (+ 1893us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 11:42:34.470535 (+ 1405us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 11:42:34.470541 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 11:42:34.470543 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0620 11:42:34.470752 (+ 209us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 11:42:34.470929 (+ 177us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 11:42:34.470930 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 11:42:34.470930 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0620 11:42:34.471025 (+ 95us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 11:42:34.471272 (+ 247us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 11:42:34.471278 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 11:42:34.471394 (+ 116us) client_negotiation.cc:769] Sending connection context
0620 11:42:34.471490 (+ 96us) client_negotiation.cc:240] Negotiation successful
0620 11:42:34.471606 (+ 116us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":133,"thread_start_us":88,"threads_started":1}
I20260620 11:42:34.472383 9010 heartbeater.cc:347] Connected to a master server at 127.6.234.254:36185
I20260620 11:42:34.472465 9010 heartbeater.cc:464] Registering TS with master...
I20260620 11:42:34.472642 9010 heartbeater.cc:511] Master 127.6.234.254:36185 requested a full tablet report, sending...
I20260620 11:42:34.473162 8807 ts_manager.cc:194] Registered new tserver with Master: 47814c4c6447409dbab77568f6434cbd (127.6.234.193:40543)
I20260620 11:42:34.473930 8807 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.6.234.193@KRBTEST.COM'} at 127.6.234.193:58833
I20260620 11:42:34.474843 7083 external_mini_cluster.cc:1388] Started /tmp/dist-test-task1JRg7N/build/release/bin/kudu as pid 8879
I20260620 11:42:34.474936 7083 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task1JRg7N/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781955746769051-7083-0/minicluster-data/ts-0/wal/instance
I20260620 11:42:34.475114 7083 external_mini_cluster.cc:1428] Setting key 77758904e5b54107e81248ebcbbcf06f
I20260620 11:42:34.480249 8863 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 11:42:34.473388 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.6.234.193:41779 (local address 127.6.234.254:36185)
0620 11:42:34.473440 (+ 52us) server_negotiation.cc:206] Beginning negotiation
0620 11:42:34.473441 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0620 11:42:34.474221 (+ 780us) server_negotiation.cc:407] Connection header received
0620 11:42:34.474264 (+ 43us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 11:42:34.474265 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 11:42:34.474274 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 11:42:34.474324 (+ 50us) server_negotiation.cc:226] Negotiated authn=SASL
0620 11:42:34.474720 (+ 396us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 11:42:34.475336 (+ 616us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 11:42:34.476298 (+ 962us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 11:42:34.476485 (+ 187us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 11:42:34.476540 (+ 55us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 11:42:34.477451 (+ 911us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 11:42:34.477452 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 11:42:34.477452 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 11:42:34.477459 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0620 11:42:34.479045 (+ 1586us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 11:42:34.479436 (+ 391us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 11:42:34.479437 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 11:42:34.479438 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 11:42:34.479499 (+ 61us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 11:42:34.479692 (+ 193us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 11:42:34.479693 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 11:42:34.479693 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 11:42:34.479804 (+ 111us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 11:42:34.479918 (+ 114us) server_negotiation.cc:1035] Waiting for connection context
0620 11:42:34.480072 (+ 154us) server_negotiation.cc:299] Negotiation successful
0620 11:42:34.480156 (+ 84us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":32}
I20260620 11:42:34.480254 9016 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 11:42:34.473381 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.234.254:36185 (local address 127.6.234.193:41779)
0620 11:42:34.474139 (+ 758us) negotiation.cc:107] Waiting for socket to connect
0620 11:42:34.474151 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0620 11:42:34.474213 (+ 62us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 11:42:34.474339 (+ 126us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 11:42:34.474340 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 11:42:34.474530 (+ 190us) client_negotiation.cc:189] Negotiated authn=SASL
0620 11:42:34.474583 (+ 53us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 11:42:34.474584 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 11:42:34.475435 (+ 851us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 11:42:34.475436 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 11:42:34.476215 (+ 779us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 11:42:34.476218 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 11:42:34.476560 (+ 342us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 11:42:34.476561 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 11:42:34.476643 (+ 82us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 11:42:34.477046 (+ 403us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 11:42:34.477052 (+ 6us) client_negotiation.cc:656] calling sasl_client_start()
0620 11:42:34.477353 (+ 301us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 11:42:34.479164 (+ 1811us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 11:42:34.479165 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 11:42:34.479166 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0620 11:42:34.479338 (+ 172us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 11:42:34.479565 (+ 227us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 11:42:34.479566 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 11:42:34.479566 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0620 11:42:34.479627 (+ 61us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 11:42:34.479913 (+ 286us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 11:42:34.479914 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 11:42:34.479962 (+ 48us) client_negotiation.cc:769] Sending connection context
0620 11:42:34.480048 (+ 86us) client_negotiation.cc:240] Negotiation successful
0620 11:42:34.480153 (+ 105us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":716,"thread_start_us":61,"threads_started":1}
WARNING: no policy specified for kudu/127.6.234.194@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.6.234.194@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.6.234.194 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task1JRg7N/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781955746769051-7083-0/krb5kdc/kudu_127.6.234.194.keytab.
Entry for principal kudu/127.6.234.194 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task1JRg7N/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781955746769051-7083-0/krb5kdc/kudu_127.6.234.194.keytab.
I20260620 11:42:34.500511 7083 external_mini_cluster.cc:1326] Running /tmp/dist-test-task1JRg7N/build/release/bin/kudu
/tmp/dist-test-task1JRg7N/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task1JRg7N/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781955746769051-7083-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-task1JRg7N/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781955746769051-7083-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task1JRg7N/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781955746769051-7083-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-task1JRg7N/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781955746769051-7083-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.6.234.194:0
--local_ip_for_outbound_sockets=127.6.234.194
--webserver_interface=127.6.234.194
--webserver_port=0
--tserver_master_addrs=127.6.234.254:36185
--keytab_file=/tmp/dist-test-task1JRg7N/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781955746769051-7083-0/krb5kdc/kudu_127.6.234.194.keytab
--principal=kudu/127.6.234.194
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.6.234.212:37773
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true
--txn_keepalive_interval_ms=500 with env {KRB5CCNAME=/tmp/dist-test-task1JRg7N/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781955746769051-7083-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task1JRg7N/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781955746769051-7083-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task1JRg7N/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781955746769051-7083-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 11:42:34.580214 9019 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 11:42:34.580418 9019 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 11:42:34.580444 9019 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 11:42:34.581578 9019 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 11:42:34.581616 9019 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 11:42:34.581636 9019 flags.cc:425] Enabled experimental flag: --txn_keepalive_interval_ms=500
W20260620 11:42:34.581648 9019 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260620 11:42:34.581666 9019 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.6.234.194
I20260620 11:42:34.583097 9019 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.6.234.212:37773
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task1JRg7N/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781955746769051-7083-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-task1JRg7N/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781955746769051-7083-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task1JRg7N/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781955746769051-7083-0/krb5kdc/kudu_127.6.234.194.keytab
--principal=kudu/127.6.234.194
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.6.234.194:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task1JRg7N/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781955746769051-7083-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.6.234.194
--webserver_port=0
--txn_keepalive_interval_ms=500
--enable_txn_system_client_init=true
--tserver_master_addrs=127.6.234.254:36185
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.9019
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.6.234.194
--log_dir=/tmp/dist-test-task1JRg7N/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781955746769051-7083-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 20 Jun 2026 10:32:37 UTC on ccycloud.quasar-bgyofv.root.comops.site
build id 4457173
I20260620 11:42:34.583283 9019 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 11:42:34.583447 9019 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 11:42:34.588361 9025 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 11:42:34.588366 9027 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 11:42:34.588330 9024 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 11:42:34.588615 9019 server_base.cc:1034] running on GCE node
I20260620 11:42:34.588842 9019 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 11:42:34.589077 9019 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 11:42:34.590214 9019 hybrid_clock.cc:648] HybridClock initialized: now 1781955754590201 us; error 27 us; skew 500 ppm
Jun 20 11:42:34 dist-test-slave-cjc6 krb5kdc[8762](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781955754, etypes {rep=17 tkt=17 ses=17}, kudu/127.6.234.194@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 11:42:34.592851 9019 init.cc:377] Logged in from keytab as kudu/127.6.234.194@KRBTEST.COM (short username kudu)
I20260620 11:42:34.593236 9019 webserver.cc:460] Webserver started at http://127.6.234.194:39579/ using document root <none> and password file <none>
I20260620 11:42:34.593472 9019 fs_manager.cc:359] Metadata directory not provided
I20260620 11:42:34.593516 9019 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 11:42:34.593605 9019 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 11:42:34.594679 9019 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task1JRg7N/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781955746769051-7083-0/minicluster-data/ts-1/data/instance:
uuid: "170e23b744f94a62854cb7cd24ac7506"
format_stamp: "Formatted at 2026-06-20 11:42:34 on dist-test-slave-cjc6"
server_key: "1967f78841e7c10fb573dca5ac403502"
server_key_iv: "40fd964606fbf143a3e3625e3eae2193"
server_key_version: "encryptionkey@0"
I20260620 11:42:34.594988 9019 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task1JRg7N/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781955746769051-7083-0/minicluster-data/ts-1/wal/instance:
uuid: "170e23b744f94a62854cb7cd24ac7506"
format_stamp: "Formatted at 2026-06-20 11:42:34 on dist-test-slave-cjc6"
server_key: "1967f78841e7c10fb573dca5ac403502"
server_key_iv: "40fd964606fbf143a3e3625e3eae2193"
server_key_version: "encryptionkey@0"
I20260620 11:42:34.596215 9019 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260620 11:42:34.596872 9034 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 11:42:34.597087 9019 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260620 11:42:34.597180 9019 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task1JRg7N/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781955746769051-7083-0/minicluster-data/ts-1/data,/tmp/dist-test-task1JRg7N/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781955746769051-7083-0/minicluster-data/ts-1/wal
uuid: "170e23b744f94a62854cb7cd24ac7506"
format_stamp: "Formatted at 2026-06-20 11:42:34 on dist-test-slave-cjc6"
server_key: "1967f78841e7c10fb573dca5ac403502"
server_key_iv: "40fd964606fbf143a3e3625e3eae2193"
server_key_version: "encryptionkey@0"
I20260620 11:42:34.597249 9019 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task1JRg7N/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781955746769051-7083-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-task1JRg7N/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781955746769051-7083-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-task1JRg7N/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781955746769051-7083-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 11:42:34.605484 9019 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 11:42:34.608119 9019 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 11:42:34.608290 9019 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 11:42:34.608793 9019 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 11:42:34.608858 9019 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 11:42:34.608903 9019 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 11:42:34.608932 9019 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 11:42:34.615680 9019 rpc_server.cc:307] RPC server started. Bound to: 127.6.234.194:44189
I20260620 11:42:34.615757 9149 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.6.234.194:44189 every 8 connection(s)
Jun 20 11:42:34 dist-test-slave-cjc6 krb5kdc[8762](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781955754, etypes {rep=17 tkt=17 ses=17}, kudu/127.6.234.194@KRBTEST.COM for kudu/127.6.234.254@KRBTEST.COM
I20260620 11:42:34.616122 9019 server_base.cc:1166] Dumped server information to /tmp/dist-test-task1JRg7N/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781955746769051-7083-0/minicluster-data/ts-1/data/info.pb
I20260620 11:42:34.619109 8863 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 11:42:34.609305 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.6.234.194:55545 (local address 127.6.234.254:36185)
0620 11:42:34.609379 (+ 74us) server_negotiation.cc:206] Beginning negotiation
0620 11:42:34.609381 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 11:42:34.609555 (+ 174us) server_negotiation.cc:407] Connection header received
0620 11:42:34.610095 (+ 540us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 11:42:34.610096 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 11:42:34.610110 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 11:42:34.610199 (+ 89us) server_negotiation.cc:226] Negotiated authn=SASL
0620 11:42:34.611025 (+ 826us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 11:42:34.611874 (+ 849us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 11:42:34.613470 (+ 1596us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 11:42:34.613826 (+ 356us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 11:42:34.613903 (+ 77us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 11:42:34.616461 (+ 2558us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 11:42:34.616463 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 11:42:34.616464 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 11:42:34.616472 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0620 11:42:34.617672 (+ 1200us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 11:42:34.618122 (+ 450us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 11:42:34.618124 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 11:42:34.618124 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 11:42:34.618183 (+ 59us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 11:42:34.618445 (+ 262us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 11:42:34.618446 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 11:42:34.618446 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 11:42:34.618570 (+ 124us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 11:42:34.618648 (+ 78us) server_negotiation.cc:1035] Waiting for connection context
0620 11:42:34.618928 (+ 280us) server_negotiation.cc:299] Negotiation successful
0620 11:42:34.619020 (+ 92us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":41}
I20260620 11:42:34.619285 9043 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 11:42:34.609261 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.234.254:36185 (local address 127.6.234.194:55545)
0620 11:42:34.609415 (+ 154us) negotiation.cc:107] Waiting for socket to connect
0620 11:42:34.609454 (+ 39us) client_negotiation.cc:174] Beginning negotiation
0620 11:42:34.609998 (+ 544us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 11:42:34.610276 (+ 278us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 11:42:34.610280 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 11:42:34.610634 (+ 354us) client_negotiation.cc:189] Negotiated authn=SASL
0620 11:42:34.610916 (+ 282us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 11:42:34.610922 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 11:42:34.611979 (+ 1057us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 11:42:34.611980 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 11:42:34.613374 (+ 1394us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 11:42:34.613377 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 11:42:34.613910 (+ 533us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 11:42:34.613911 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 11:42:34.614054 (+ 143us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 11:42:34.614594 (+ 540us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 11:42:34.614604 (+ 10us) client_negotiation.cc:656] calling sasl_client_start()
0620 11:42:34.616359 (+ 1755us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 11:42:34.617813 (+ 1454us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 11:42:34.617820 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 11:42:34.617822 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0620 11:42:34.618041 (+ 219us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 11:42:34.618259 (+ 218us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 11:42:34.618260 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 11:42:34.618260 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0620 11:42:34.618375 (+ 115us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 11:42:34.618638 (+ 263us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 11:42:34.618645 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 11:42:34.618782 (+ 137us) client_negotiation.cc:769] Sending connection context
0620 11:42:34.618903 (+ 121us) client_negotiation.cc:240] Negotiation successful
0620 11:42:34.619060 (+ 157us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":110,"thread_start_us":85,"threads_started":1}
I20260620 11:42:34.619911 9150 heartbeater.cc:347] Connected to a master server at 127.6.234.254:36185
I20260620 11:42:34.620003 9150 heartbeater.cc:464] Registering TS with master...
I20260620 11:42:34.620184 9150 heartbeater.cc:511] Master 127.6.234.254:36185 requested a full tablet report, sending...
I20260620 11:42:34.620853 8807 ts_manager.cc:194] Registered new tserver with Master: 170e23b744f94a62854cb7cd24ac7506 (127.6.234.194:44189)
I20260620 11:42:34.621351 8807 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.6.234.194@KRBTEST.COM'} at 127.6.234.194:55545
I20260620 11:42:34.625066 7083 external_mini_cluster.cc:1388] Started /tmp/dist-test-task1JRg7N/build/release/bin/kudu as pid 9019
I20260620 11:42:34.625180 7083 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task1JRg7N/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781955746769051-7083-0/minicluster-data/ts-1/wal/instance
I20260620 11:42:34.625344 7083 external_mini_cluster.cc:1428] Setting key 334ddda26bcdeb259f59f68f866a1f28
I20260620 11:42:34.626137 9156 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 11:42:34.620894 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.234.254:36185 (local address 127.6.234.194:36305)
0620 11:42:34.621059 (+ 165us) negotiation.cc:107] Waiting for socket to connect
0620 11:42:34.621071 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0620 11:42:34.621120 (+ 49us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 11:42:34.621234 (+ 114us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 11:42:34.621235 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 11:42:34.621380 (+ 145us) client_negotiation.cc:189] Negotiated authn=SASL
0620 11:42:34.621427 (+ 47us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 11:42:34.621429 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 11:42:34.622160 (+ 731us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 11:42:34.622161 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 11:42:34.622667 (+ 506us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 11:42:34.622669 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 11:42:34.623050 (+ 381us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 11:42:34.623051 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 11:42:34.623137 (+ 86us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 11:42:34.623496 (+ 359us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 11:42:34.623500 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0620 11:42:34.623758 (+ 258us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 11:42:34.624998 (+ 1240us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 11:42:34.624999 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 11:42:34.625000 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0620 11:42:34.625163 (+ 163us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 11:42:34.625399 (+ 236us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 11:42:34.625400 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 11:42:34.625400 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0620 11:42:34.625458 (+ 58us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 11:42:34.625818 (+ 360us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 11:42:34.625819 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 11:42:34.625860 (+ 41us) client_negotiation.cc:769] Sending connection context
0620 11:42:34.625974 (+ 114us) client_negotiation.cc:240] Negotiation successful
0620 11:42:34.626074 (+ 100us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":135,"thread_start_us":71,"threads_started":1}
I20260620 11:42:34.626334 8863 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 11:42:34.620986 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.6.234.194:36305 (local address 127.6.234.254:36185)
0620 11:42:34.621029 (+ 43us) server_negotiation.cc:206] Beginning negotiation
0620 11:42:34.621031 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 11:42:34.621128 (+ 97us) server_negotiation.cc:407] Connection header received
0620 11:42:34.621152 (+ 24us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 11:42:34.621153 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 11:42:34.621162 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 11:42:34.621220 (+ 58us) server_negotiation.cc:226] Negotiated authn=SASL
0620 11:42:34.621516 (+ 296us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 11:42:34.622065 (+ 549us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 11:42:34.622763 (+ 698us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 11:42:34.622965 (+ 202us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 11:42:34.623048 (+ 83us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 11:42:34.623845 (+ 797us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 11:42:34.623846 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 11:42:34.623847 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 11:42:34.623853 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0620 11:42:34.624908 (+ 1055us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 11:42:34.625271 (+ 363us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 11:42:34.625272 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 11:42:34.625273 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 11:42:34.625324 (+ 51us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 11:42:34.625541 (+ 217us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 11:42:34.625542 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 11:42:34.625542 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 11:42:34.625747 (+ 205us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 11:42:34.625815 (+ 68us) server_negotiation.cc:1035] Waiting for connection context
0620 11:42:34.626133 (+ 318us) server_negotiation.cc:299] Negotiation successful
0620 11:42:34.626212 (+ 79us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":20}
WARNING: no policy specified for kudu/127.6.234.195@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.6.234.195@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.6.234.195 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task1JRg7N/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781955746769051-7083-0/krb5kdc/kudu_127.6.234.195.keytab.
Entry for principal kudu/127.6.234.195 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task1JRg7N/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781955746769051-7083-0/krb5kdc/kudu_127.6.234.195.keytab.
I20260620 11:42:34.650031 7083 external_mini_cluster.cc:1326] Running /tmp/dist-test-task1JRg7N/build/release/bin/kudu
/tmp/dist-test-task1JRg7N/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task1JRg7N/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781955746769051-7083-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-task1JRg7N/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781955746769051-7083-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task1JRg7N/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781955746769051-7083-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-task1JRg7N/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781955746769051-7083-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.6.234.195:0
--local_ip_for_outbound_sockets=127.6.234.195
--webserver_interface=127.6.234.195
--webserver_port=0
--tserver_master_addrs=127.6.234.254:36185
--keytab_file=/tmp/dist-test-task1JRg7N/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781955746769051-7083-0/krb5kdc/kudu_127.6.234.195.keytab
--principal=kudu/127.6.234.195
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.6.234.212:37773
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true
--txn_keepalive_interval_ms=500 with env {KRB5CCNAME=/tmp/dist-test-task1JRg7N/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781955746769051-7083-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task1JRg7N/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781955746769051-7083-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task1JRg7N/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781955746769051-7083-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 11:42:34.731472 9159 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 11:42:34.731699 9159 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 11:42:34.731740 9159 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 11:42:34.732903 9159 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 11:42:34.732972 9159 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 11:42:34.733008 9159 flags.cc:425] Enabled experimental flag: --txn_keepalive_interval_ms=500
W20260620 11:42:34.733055 9159 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260620 11:42:34.733094 9159 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.6.234.195
I20260620 11:42:34.734580 9159 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.6.234.212:37773
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task1JRg7N/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781955746769051-7083-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-task1JRg7N/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781955746769051-7083-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task1JRg7N/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781955746769051-7083-0/krb5kdc/kudu_127.6.234.195.keytab
--principal=kudu/127.6.234.195
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.6.234.195:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task1JRg7N/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781955746769051-7083-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.6.234.195
--webserver_port=0
--txn_keepalive_interval_ms=500
--enable_txn_system_client_init=true
--tserver_master_addrs=127.6.234.254:36185
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.9159
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.6.234.195
--log_dir=/tmp/dist-test-task1JRg7N/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781955746769051-7083-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 20 Jun 2026 10:32:37 UTC on ccycloud.quasar-bgyofv.root.comops.site
build id 4457173
I20260620 11:42:34.734848 9159 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 11:42:34.735078 9159 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 11:42:34.740083 9167 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 11:42:34.740162 9159 server_base.cc:1034] running on GCE node
W20260620 11:42:34.741595 9165 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 11:42:34.742070 9164 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 11:42:34.742316 9159 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 11:42:34.742568 9159 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 11:42:34.743716 9159 hybrid_clock.cc:648] HybridClock initialized: now 1781955754743696 us; error 30 us; skew 500 ppm
Jun 20 11:42:34 dist-test-slave-cjc6 krb5kdc[8762](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781955754, etypes {rep=17 tkt=17 ses=17}, kudu/127.6.234.195@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 11:42:34.746299 9159 init.cc:377] Logged in from keytab as kudu/127.6.234.195@KRBTEST.COM (short username kudu)
I20260620 11:42:34.746722 9159 webserver.cc:460] Webserver started at http://127.6.234.195:34489/ using document root <none> and password file <none>
I20260620 11:42:34.746919 9159 fs_manager.cc:359] Metadata directory not provided
I20260620 11:42:34.746961 9159 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 11:42:34.747045 9159 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 11:42:34.748008 9159 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task1JRg7N/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781955746769051-7083-0/minicluster-data/ts-2/data/instance:
uuid: "36f86b86532e42758e1f5249c186f526"
format_stamp: "Formatted at 2026-06-20 11:42:34 on dist-test-slave-cjc6"
server_key: "934b4864cf8d2d4b1963b82ed45dcda3"
server_key_iv: "40eff8a54643d4e577a65b5a11497ac1"
server_key_version: "encryptionkey@0"
I20260620 11:42:34.748291 9159 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task1JRg7N/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781955746769051-7083-0/minicluster-data/ts-2/wal/instance:
uuid: "36f86b86532e42758e1f5249c186f526"
format_stamp: "Formatted at 2026-06-20 11:42:34 on dist-test-slave-cjc6"
server_key: "934b4864cf8d2d4b1963b82ed45dcda3"
server_key_iv: "40eff8a54643d4e577a65b5a11497ac1"
server_key_version: "encryptionkey@0"
I20260620 11:42:34.749600 9159 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.001s
I20260620 11:42:34.750398 9174 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 11:42:34.750607 9159 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.001s
I20260620 11:42:34.750696 9159 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task1JRg7N/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781955746769051-7083-0/minicluster-data/ts-2/data,/tmp/dist-test-task1JRg7N/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781955746769051-7083-0/minicluster-data/ts-2/wal
uuid: "36f86b86532e42758e1f5249c186f526"
format_stamp: "Formatted at 2026-06-20 11:42:34 on dist-test-slave-cjc6"
server_key: "934b4864cf8d2d4b1963b82ed45dcda3"
server_key_iv: "40eff8a54643d4e577a65b5a11497ac1"
server_key_version: "encryptionkey@0"
I20260620 11:42:34.750766 9159 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task1JRg7N/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781955746769051-7083-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-task1JRg7N/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781955746769051-7083-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-task1JRg7N/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781955746769051-7083-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 11:42:34.764555 9159 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 11:42:34.766877 9159 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 11:42:34.767042 9159 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 11:42:34.767534 9159 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 11:42:34.767570 9159 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 11:42:34.767591 9159 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 11:42:34.767642 9159 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Jun 20 11:42:34 dist-test-slave-cjc6 krb5kdc[8762](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781955754, etypes {rep=17 tkt=17 ses=17}, kudu/127.6.234.195@KRBTEST.COM for kudu/127.6.234.254@KRBTEST.COM
I20260620 11:42:34.774536 9159 rpc_server.cc:307] RPC server started. Bound to: 127.6.234.195:37197
I20260620 11:42:34.774583 9289 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.6.234.195:37197 every 8 connection(s)
I20260620 11:42:34.774945 9159 server_base.cc:1166] Dumped server information to /tmp/dist-test-task1JRg7N/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781955746769051-7083-0/minicluster-data/ts-2/data/info.pb
I20260620 11:42:34.777307 8863 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 11:42:34.768147 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.6.234.195:35309 (local address 127.6.234.254:36185)
0620 11:42:34.768196 (+ 49us) server_negotiation.cc:206] Beginning negotiation
0620 11:42:34.768198 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 11:42:34.768443 (+ 245us) server_negotiation.cc:407] Connection header received
0620 11:42:34.769026 (+ 583us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 11:42:34.769028 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 11:42:34.769046 (+ 18us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 11:42:34.769122 (+ 76us) server_negotiation.cc:226] Negotiated authn=SASL
0620 11:42:34.770037 (+ 915us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 11:42:34.770535 (+ 498us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 11:42:34.772131 (+ 1596us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 11:42:34.772479 (+ 348us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 11:42:34.772576 (+ 97us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 11:42:34.774839 (+ 2263us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 11:42:34.774840 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 11:42:34.774840 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 11:42:34.774849 (+ 9us) server_negotiation.cc:922] calling sasl_server_start()
0620 11:42:34.776231 (+ 1382us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 11:42:34.776581 (+ 350us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 11:42:34.776582 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 11:42:34.776583 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 11:42:34.776631 (+ 48us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 11:42:34.776807 (+ 176us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 11:42:34.776807 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 11:42:34.776808 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 11:42:34.776907 (+ 99us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 11:42:34.776984 (+ 77us) server_negotiation.cc:1035] Waiting for connection context
0620 11:42:34.777186 (+ 202us) server_negotiation.cc:299] Negotiation successful
0620 11:42:34.777245 (+ 59us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":23}
I20260620 11:42:34.777549 9183 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 11:42:34.768125 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.234.254:36185 (local address 127.6.234.195:35309)
0620 11:42:34.768305 (+ 180us) negotiation.cc:107] Waiting for socket to connect
0620 11:42:34.768347 (+ 42us) client_negotiation.cc:174] Beginning negotiation
0620 11:42:34.768858 (+ 511us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 11:42:34.769237 (+ 379us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 11:42:34.769242 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 11:42:34.769609 (+ 367us) client_negotiation.cc:189] Negotiated authn=SASL
0620 11:42:34.769920 (+ 311us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 11:42:34.769927 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 11:42:34.770626 (+ 699us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 11:42:34.770627 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 11:42:34.772005 (+ 1378us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 11:42:34.772009 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 11:42:34.772576 (+ 567us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 11:42:34.772577 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 11:42:34.772691 (+ 114us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 11:42:34.773216 (+ 525us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 11:42:34.773242 (+ 26us) client_negotiation.cc:656] calling sasl_client_start()
0620 11:42:34.774741 (+ 1499us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 11:42:34.776296 (+ 1555us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 11:42:34.776300 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 11:42:34.776302 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0620 11:42:34.776514 (+ 212us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 11:42:34.776679 (+ 165us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 11:42:34.776680 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 11:42:34.776680 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0620 11:42:34.776759 (+ 79us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 11:42:34.776982 (+ 223us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 11:42:34.776986 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 11:42:34.777092 (+ 106us) client_negotiation.cc:769] Sending connection context
0620 11:42:34.777172 (+ 80us) client_negotiation.cc:240] Negotiation successful
0620 11:42:34.777284 (+ 112us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":142,"thread_start_us":99,"threads_started":1}
I20260620 11:42:34.778085 9290 heartbeater.cc:347] Connected to a master server at 127.6.234.254:36185
I20260620 11:42:34.778168 9290 heartbeater.cc:464] Registering TS with master...
I20260620 11:42:34.778359 9290 heartbeater.cc:511] Master 127.6.234.254:36185 requested a full tablet report, sending...
I20260620 11:42:34.778689 8807 ts_manager.cc:194] Registered new tserver with Master: 36f86b86532e42758e1f5249c186f526 (127.6.234.195:37197)
I20260620 11:42:34.779152 8807 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.6.234.195@KRBTEST.COM'} at 127.6.234.195:35309
I20260620 11:42:34.783824 8863 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 11:42:34.779183 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.6.234.195:58611 (local address 127.6.234.254:36185)
0620 11:42:34.779227 (+ 44us) server_negotiation.cc:206] Beginning negotiation
0620 11:42:34.779229 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 11:42:34.779342 (+ 113us) server_negotiation.cc:407] Connection header received
0620 11:42:34.779404 (+ 62us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 11:42:34.779405 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 11:42:34.779416 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 11:42:34.779477 (+ 61us) server_negotiation.cc:226] Negotiated authn=SASL
0620 11:42:34.779825 (+ 348us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 11:42:34.780263 (+ 438us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 11:42:34.780894 (+ 631us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 11:42:34.781068 (+ 174us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 11:42:34.781120 (+ 52us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 11:42:34.781939 (+ 819us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 11:42:34.781940 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 11:42:34.781940 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 11:42:34.781945 (+ 5us) server_negotiation.cc:922] calling sasl_server_start()
0620 11:42:34.783026 (+ 1081us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 11:42:34.783302 (+ 276us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 11:42:34.783303 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 11:42:34.783303 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 11:42:34.783341 (+ 38us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 11:42:34.783498 (+ 157us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 11:42:34.783498 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 11:42:34.783499 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 11:42:34.783577 (+ 78us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 11:42:34.783619 (+ 42us) server_negotiation.cc:1035] Waiting for connection context
0620 11:42:34.783719 (+ 100us) server_negotiation.cc:299] Negotiation successful
0620 11:42:34.783769 (+ 50us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":24}
I20260620 11:42:34.783842 9296 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 11:42:34.779122 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.234.254:36185 (local address 127.6.234.195:58611)
0620 11:42:34.779271 (+ 149us) negotiation.cc:107] Waiting for socket to connect
0620 11:42:34.779281 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0620 11:42:34.779336 (+ 55us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 11:42:34.779489 (+ 153us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 11:42:34.779490 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 11:42:34.779686 (+ 196us) client_negotiation.cc:189] Negotiated authn=SASL
0620 11:42:34.779751 (+ 65us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 11:42:34.779752 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 11:42:34.780349 (+ 597us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 11:42:34.780350 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 11:42:34.780829 (+ 479us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 11:42:34.780830 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 11:42:34.781133 (+ 303us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 11:42:34.781134 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 11:42:34.781186 (+ 52us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 11:42:34.781528 (+ 342us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 11:42:34.781532 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0620 11:42:34.781877 (+ 345us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 11:42:34.783092 (+ 1215us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 11:42:34.783093 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 11:42:34.783093 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0620 11:42:34.783247 (+ 154us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 11:42:34.783391 (+ 144us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 11:42:34.783392 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 11:42:34.783392 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0620 11:42:34.783452 (+ 60us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 11:42:34.783624 (+ 172us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 11:42:34.783625 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 11:42:34.783658 (+ 33us) client_negotiation.cc:769] Sending connection context
0620 11:42:34.783706 (+ 48us) client_negotiation.cc:240] Negotiation successful
0620 11:42:34.783775 (+ 69us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":123,"thread_start_us":64,"threads_started":1}
I20260620 11:42:34.784734 7083 external_mini_cluster.cc:1388] Started /tmp/dist-test-task1JRg7N/build/release/bin/kudu as pid 9159
I20260620 11:42:34.784811 7083 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task1JRg7N/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781955746769051-7083-0/minicluster-data/ts-2/wal/instance
I20260620 11:42:34.785045 7083 external_mini_cluster.cc:1428] Setting key b961624ee5a7076133499204fe77e789
I20260620 11:42:34.785912 7083 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
Jun 20 11:42:34 dist-test-slave-cjc6 krb5kdc[8762](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781955754, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
Jun 20 11:42:34 dist-test-slave-cjc6 krb5kdc[8762](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781955754, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.6.234.254@KRBTEST.COM
I20260620 11:42:34.808050 8863 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 11:42:34.801204 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47676 (local address 127.6.234.254:36185)
0620 11:42:34.801264 (+ 60us) server_negotiation.cc:206] Beginning negotiation
0620 11:42:34.801266 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 11:42:34.801279 (+ 13us) server_negotiation.cc:407] Connection header received
0620 11:42:34.801350 (+ 71us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 11:42:34.801351 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 11:42:34.801360 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 11:42:34.801411 (+ 51us) server_negotiation.cc:226] Negotiated authn=SASL
0620 11:42:34.801998 (+ 587us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 11:42:34.802475 (+ 477us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 11:42:34.803156 (+ 681us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 11:42:34.803354 (+ 198us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 11:42:34.803420 (+ 66us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 11:42:34.805585 (+ 2165us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 11:42:34.805588 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 11:42:34.805588 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 11:42:34.805597 (+ 9us) server_negotiation.cc:922] calling sasl_server_start()
0620 11:42:34.806820 (+ 1223us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 11:42:34.807241 (+ 421us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 11:42:34.807242 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 11:42:34.807243 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 11:42:34.807300 (+ 57us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 11:42:34.807509 (+ 209us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 11:42:34.807510 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 11:42:34.807510 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 11:42:34.807623 (+ 113us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 11:42:34.807733 (+ 110us) server_negotiation.cc:1035] Waiting for connection context
0620 11:42:34.807854 (+ 121us) server_negotiation.cc:299] Negotiation successful
0620 11:42:34.807947 (+ 93us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":38}
I20260620 11:42:34.809599 8807 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:47676:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260620 11:42:34.811182 8807 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260620 11:42:34.817497 9304 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 11:42:34.813259 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.234.193:40543 (local address 127.0.0.1:58340)
0620 11:42:34.813551 (+ 292us) negotiation.cc:107] Waiting for socket to connect
0620 11:42:34.813562 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0620 11:42:34.813610 (+ 48us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 11:42:34.814081 (+ 471us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 11:42:34.814082 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 11:42:34.814083 (+ 1us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 11:42:34.814123 (+ 40us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 11:42:34.814124 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 11:42:34.815054 (+ 930us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 11:42:34.815055 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 11:42:34.816234 (+ 1179us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 11:42:34.816236 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 11:42:34.817179 (+ 943us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 11:42:34.817180 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 11:42:34.817244 (+ 64us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 11:42:34.817254 (+ 10us) client_negotiation.cc:769] Sending connection context
0620 11:42:34.817307 (+ 53us) client_negotiation.cc:240] Negotiation successful
0620 11:42:34.817377 (+ 70us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":278,"mutex_wait_us":3,"thread_start_us":52,"threads_started":1}
I20260620 11:42:34.817646 9307 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 11:42:34.813414 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45108 (local address 127.6.234.195:37197)
0620 11:42:34.813841 (+ 427us) server_negotiation.cc:206] Beginning negotiation
0620 11:42:34.813845 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 11:42:34.813854 (+ 9us) server_negotiation.cc:407] Connection header received
0620 11:42:34.813894 (+ 40us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 11:42:34.813896 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 11:42:34.813912 (+ 16us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 11:42:34.814237 (+ 325us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 11:42:34.814293 (+ 56us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 11:42:34.815380 (+ 1087us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 11:42:34.816601 (+ 1221us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 11:42:34.817315 (+ 714us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 11:42:34.817375 (+ 60us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 11:42:34.817426 (+ 51us) server_negotiation.cc:1035] Waiting for connection context
0620 11:42:34.817505 (+ 79us) server_negotiation.cc:299] Negotiation successful
0620 11:42:34.817548 (+ 43us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":394,"thread_start_us":100,"threads_started":1}
I20260620 11:42:34.817688 9309 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 11:42:34.813634 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58340 (local address 127.6.234.193:40543)
0620 11:42:34.813886 (+ 252us) server_negotiation.cc:206] Beginning negotiation
0620 11:42:34.813890 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 11:42:34.813901 (+ 11us) server_negotiation.cc:407] Connection header received
0620 11:42:34.813947 (+ 46us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 11:42:34.813949 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 11:42:34.813960 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 11:42:34.814199 (+ 239us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 11:42:34.814260 (+ 61us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 11:42:34.814964 (+ 704us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 11:42:34.816319 (+ 1355us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 11:42:34.817074 (+ 755us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 11:42:34.817457 (+ 383us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 11:42:34.817525 (+ 68us) server_negotiation.cc:1035] Waiting for connection context
0620 11:42:34.817547 (+ 22us) server_negotiation.cc:299] Negotiation successful
0620 11:42:34.817584 (+ 37us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":219,"thread_start_us":134,"threads_started":1}
I20260620 11:42:34.817744 9306 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 11:42:34.813463 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.234.194:44189 (local address 127.0.0.1:53692)
0620 11:42:34.813771 (+ 308us) negotiation.cc:107] Waiting for socket to connect
0620 11:42:34.813782 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0620 11:42:34.813836 (+ 54us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 11:42:34.814042 (+ 206us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 11:42:34.814046 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 11:42:34.814048 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 11:42:34.814122 (+ 74us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 11:42:34.814124 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 11:42:34.815473 (+ 1349us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 11:42:34.815474 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 11:42:34.816397 (+ 923us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 11:42:34.816399 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 11:42:34.817148 (+ 749us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 11:42:34.817149 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 11:42:34.817237 (+ 88us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 11:42:34.817251 (+ 14us) client_negotiation.cc:769] Sending connection context
0620 11:42:34.817348 (+ 97us) client_negotiation.cc:240] Negotiation successful
0620 11:42:34.817400 (+ 52us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":296,"mutex_wait_us":13,"thread_start_us":41,"threads_started":1}
I20260620 11:42:34.818127 9305 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 11:42:34.813195 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.234.195:37197 (local address 127.0.0.1:45108)
0620 11:42:34.813361 (+ 166us) negotiation.cc:107] Waiting for socket to connect
0620 11:42:34.813399 (+ 38us) client_negotiation.cc:174] Beginning negotiation
0620 11:42:34.813467 (+ 68us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 11:42:34.814107 (+ 640us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 11:42:34.814108 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 11:42:34.814110 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 11:42:34.814157 (+ 47us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 11:42:34.814159 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 11:42:34.815463 (+ 1304us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 11:42:34.815464 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 11:42:34.816520 (+ 1056us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 11:42:34.816522 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 11:42:34.817386 (+ 864us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 11:42:34.817387 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 11:42:34.817442 (+ 55us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 11:42:34.817445 (+ 3us) client_negotiation.cc:769] Sending connection context
0620 11:42:34.817492 (+ 47us) client_negotiation.cc:240] Negotiation successful
0620 11:42:34.817529 (+ 37us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":140,"mutex_wait_us":23,"thread_start_us":70,"threads_started":1}
I20260620 11:42:34.818459 9308 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 11:42:34.813589 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53692 (local address 127.6.234.194:44189)
0620 11:42:34.813769 (+ 180us) server_negotiation.cc:206] Beginning negotiation
0620 11:42:34.813774 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 11:42:34.813827 (+ 53us) server_negotiation.cc:407] Connection header received
0620 11:42:34.813899 (+ 72us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 11:42:34.813900 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 11:42:34.813914 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 11:42:34.814012 (+ 98us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 11:42:34.814288 (+ 276us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 11:42:34.815371 (+ 1083us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 11:42:34.816481 (+ 1110us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 11:42:34.817042 (+ 561us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 11:42:34.817144 (+ 102us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 11:42:34.817207 (+ 63us) server_negotiation.cc:1035] Waiting for connection context
0620 11:42:34.818310 (+ 1103us) server_negotiation.cc:299] Negotiation successful
0620 11:42:34.818357 (+ 47us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":148,"thread_start_us":97,"threads_started":1}
I20260620 11:42:34.818356 8944 tablet_service.cc:1467] Processing CreateTablet for tablet fa7dc55b71324dd2be32fd47a168d402 (DEFAULT_TABLE table=test-table [id=52b490d05bc34359be17ea89ded471b9]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 11:42:34.818763 8944 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet fa7dc55b71324dd2be32fd47a168d402. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 11:42:34.818826 9224 tablet_service.cc:1467] Processing CreateTablet for tablet fa7dc55b71324dd2be32fd47a168d402 (DEFAULT_TABLE table=test-table [id=52b490d05bc34359be17ea89ded471b9]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 11:42:34.819023 9084 tablet_service.cc:1467] Processing CreateTablet for tablet fa7dc55b71324dd2be32fd47a168d402 (DEFAULT_TABLE table=test-table [id=52b490d05bc34359be17ea89ded471b9]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 11:42:34.819159 9224 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet fa7dc55b71324dd2be32fd47a168d402. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 11:42:34.819319 9084 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet fa7dc55b71324dd2be32fd47a168d402. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 11:42:34.822350 9310 tablet_bootstrap.cc:492] T fa7dc55b71324dd2be32fd47a168d402 P 47814c4c6447409dbab77568f6434cbd: Bootstrap starting.
I20260620 11:42:34.822775 9312 tablet_bootstrap.cc:492] T fa7dc55b71324dd2be32fd47a168d402 P 170e23b744f94a62854cb7cd24ac7506: Bootstrap starting.
I20260620 11:42:34.823086 9310 tablet_bootstrap.cc:654] T fa7dc55b71324dd2be32fd47a168d402 P 47814c4c6447409dbab77568f6434cbd: Neither blocks nor log segments found. Creating new log.
I20260620 11:42:34.823406 9312 tablet_bootstrap.cc:654] T fa7dc55b71324dd2be32fd47a168d402 P 170e23b744f94a62854cb7cd24ac7506: Neither blocks nor log segments found. Creating new log.
I20260620 11:42:34.823417 9310 log.cc:824] T fa7dc55b71324dd2be32fd47a168d402 P 47814c4c6447409dbab77568f6434cbd: Log is configured to *not* fsync() on all Append() calls
I20260620 11:42:34.823745 9312 log.cc:824] T fa7dc55b71324dd2be32fd47a168d402 P 170e23b744f94a62854cb7cd24ac7506: Log is configured to *not* fsync() on all Append() calls
I20260620 11:42:34.823768 9311 tablet_bootstrap.cc:492] T fa7dc55b71324dd2be32fd47a168d402 P 36f86b86532e42758e1f5249c186f526: Bootstrap starting.
I20260620 11:42:34.824221 9310 tablet_bootstrap.cc:492] T fa7dc55b71324dd2be32fd47a168d402 P 47814c4c6447409dbab77568f6434cbd: No bootstrap required, opened a new log
I20260620 11:42:34.824306 9310 ts_tablet_manager.cc:1397] T fa7dc55b71324dd2be32fd47a168d402 P 47814c4c6447409dbab77568f6434cbd: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260620 11:42:34.824529 9312 tablet_bootstrap.cc:492] T fa7dc55b71324dd2be32fd47a168d402 P 170e23b744f94a62854cb7cd24ac7506: No bootstrap required, opened a new log
I20260620 11:42:34.824596 9312 ts_tablet_manager.cc:1397] T fa7dc55b71324dd2be32fd47a168d402 P 170e23b744f94a62854cb7cd24ac7506: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260620 11:42:34.824920 9311 tablet_bootstrap.cc:654] T fa7dc55b71324dd2be32fd47a168d402 P 36f86b86532e42758e1f5249c186f526: Neither blocks nor log segments found. Creating new log.
I20260620 11:42:34.825251 9311 log.cc:824] T fa7dc55b71324dd2be32fd47a168d402 P 36f86b86532e42758e1f5249c186f526: Log is configured to *not* fsync() on all Append() calls
I20260620 11:42:34.826025 9311 tablet_bootstrap.cc:492] T fa7dc55b71324dd2be32fd47a168d402 P 36f86b86532e42758e1f5249c186f526: No bootstrap required, opened a new log
I20260620 11:42:34.826097 9311 ts_tablet_manager.cc:1397] T fa7dc55b71324dd2be32fd47a168d402 P 36f86b86532e42758e1f5249c186f526: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20260620 11:42:34.826278 9310 raft_consensus.cc:348] T fa7dc55b71324dd2be32fd47a168d402 P 47814c4c6447409dbab77568f6434cbd [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "36f86b86532e42758e1f5249c186f526" member_type: VOTER last_known_addr { host: "127.6.234.195" port: 37197 } } peers { permanent_uuid: "47814c4c6447409dbab77568f6434cbd" member_type: VOTER last_known_addr { host: "127.6.234.193" port: 40543 } } peers { permanent_uuid: "170e23b744f94a62854cb7cd24ac7506" member_type: VOTER last_known_addr { host: "127.6.234.194" port: 44189 } }
I20260620 11:42:34.826428 9310 raft_consensus.cc:374] T fa7dc55b71324dd2be32fd47a168d402 P 47814c4c6447409dbab77568f6434cbd [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 11:42:34.826468 9310 raft_consensus.cc:729] T fa7dc55b71324dd2be32fd47a168d402 P 47814c4c6447409dbab77568f6434cbd [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 47814c4c6447409dbab77568f6434cbd, State: Initialized, Role: FOLLOWER
I20260620 11:42:34.826603 9310 consensus_queue.cc:260] T fa7dc55b71324dd2be32fd47a168d402 P 47814c4c6447409dbab77568f6434cbd [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "36f86b86532e42758e1f5249c186f526" member_type: VOTER last_known_addr { host: "127.6.234.195" port: 37197 } } peers { permanent_uuid: "47814c4c6447409dbab77568f6434cbd" member_type: VOTER last_known_addr { host: "127.6.234.193" port: 40543 } } peers { permanent_uuid: "170e23b744f94a62854cb7cd24ac7506" member_type: VOTER last_known_addr { host: "127.6.234.194" port: 44189 } }
I20260620 11:42:34.826668 9312 raft_consensus.cc:348] T fa7dc55b71324dd2be32fd47a168d402 P 170e23b744f94a62854cb7cd24ac7506 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "36f86b86532e42758e1f5249c186f526" member_type: VOTER last_known_addr { host: "127.6.234.195" port: 37197 } } peers { permanent_uuid: "47814c4c6447409dbab77568f6434cbd" member_type: VOTER last_known_addr { host: "127.6.234.193" port: 40543 } } peers { permanent_uuid: "170e23b744f94a62854cb7cd24ac7506" member_type: VOTER last_known_addr { host: "127.6.234.194" port: 44189 } }
I20260620 11:42:34.826828 9312 raft_consensus.cc:374] T fa7dc55b71324dd2be32fd47a168d402 P 170e23b744f94a62854cb7cd24ac7506 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 11:42:34.826875 9312 raft_consensus.cc:729] T fa7dc55b71324dd2be32fd47a168d402 P 170e23b744f94a62854cb7cd24ac7506 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 170e23b744f94a62854cb7cd24ac7506, State: Initialized, Role: FOLLOWER
I20260620 11:42:34.826874 9310 ts_tablet_manager.cc:1428] T fa7dc55b71324dd2be32fd47a168d402 P 47814c4c6447409dbab77568f6434cbd: Time spent starting tablet: real 0.003s user 0.003s sys 0.000s
I20260620 11:42:34.827009 9010 heartbeater.cc:503] Master 127.6.234.254:36185 was elected leader, sending a full tablet report...
I20260620 11:42:34.827011 9312 consensus_queue.cc:260] T fa7dc55b71324dd2be32fd47a168d402 P 170e23b744f94a62854cb7cd24ac7506 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "36f86b86532e42758e1f5249c186f526" member_type: VOTER last_known_addr { host: "127.6.234.195" port: 37197 } } peers { permanent_uuid: "47814c4c6447409dbab77568f6434cbd" member_type: VOTER last_known_addr { host: "127.6.234.193" port: 40543 } } peers { permanent_uuid: "170e23b744f94a62854cb7cd24ac7506" member_type: VOTER last_known_addr { host: "127.6.234.194" port: 44189 } }
I20260620 11:42:34.827266 9312 ts_tablet_manager.cc:1428] T fa7dc55b71324dd2be32fd47a168d402 P 170e23b744f94a62854cb7cd24ac7506: Time spent starting tablet: real 0.003s user 0.003s sys 0.000s
I20260620 11:42:34.827307 9150 heartbeater.cc:503] Master 127.6.234.254:36185 was elected leader, sending a full tablet report...
I20260620 11:42:34.828361 9311 raft_consensus.cc:348] T fa7dc55b71324dd2be32fd47a168d402 P 36f86b86532e42758e1f5249c186f526 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "36f86b86532e42758e1f5249c186f526" member_type: VOTER last_known_addr { host: "127.6.234.195" port: 37197 } } peers { permanent_uuid: "47814c4c6447409dbab77568f6434cbd" member_type: VOTER last_known_addr { host: "127.6.234.193" port: 40543 } } peers { permanent_uuid: "170e23b744f94a62854cb7cd24ac7506" member_type: VOTER last_known_addr { host: "127.6.234.194" port: 44189 } }
I20260620 11:42:34.828471 9311 raft_consensus.cc:374] T fa7dc55b71324dd2be32fd47a168d402 P 36f86b86532e42758e1f5249c186f526 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 11:42:34.828500 9311 raft_consensus.cc:729] T fa7dc55b71324dd2be32fd47a168d402 P 36f86b86532e42758e1f5249c186f526 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 36f86b86532e42758e1f5249c186f526, State: Initialized, Role: FOLLOWER
I20260620 11:42:34.828617 9311 consensus_queue.cc:260] T fa7dc55b71324dd2be32fd47a168d402 P 36f86b86532e42758e1f5249c186f526 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "36f86b86532e42758e1f5249c186f526" member_type: VOTER last_known_addr { host: "127.6.234.195" port: 37197 } } peers { permanent_uuid: "47814c4c6447409dbab77568f6434cbd" member_type: VOTER last_known_addr { host: "127.6.234.193" port: 40543 } } peers { permanent_uuid: "170e23b744f94a62854cb7cd24ac7506" member_type: VOTER last_known_addr { host: "127.6.234.194" port: 44189 } }
I20260620 11:42:34.828890 9311 ts_tablet_manager.cc:1428] T fa7dc55b71324dd2be32fd47a168d402 P 36f86b86532e42758e1f5249c186f526: Time spent starting tablet: real 0.003s user 0.003s sys 0.000s
I20260620 11:42:34.828925 9290 heartbeater.cc:503] Master 127.6.234.254:36185 was elected leader, sending a full tablet report...
W20260620 11:42:34.867033 9151 tablet.cc:2369] T fa7dc55b71324dd2be32fd47a168d402 P 170e23b744f94a62854cb7cd24ac7506: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260620 11:42:34.969480 9011 tablet.cc:2369] T fa7dc55b71324dd2be32fd47a168d402 P 47814c4c6447409dbab77568f6434cbd: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260620 11:42:35.025569 9291 tablet.cc:2369] T fa7dc55b71324dd2be32fd47a168d402 P 36f86b86532e42758e1f5249c186f526: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260620 11:42:35.147877 9318 raft_consensus.cc:482] T fa7dc55b71324dd2be32fd47a168d402 P 36f86b86532e42758e1f5249c186f526 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260620 11:42:35.148015 9318 raft_consensus.cc:504] T fa7dc55b71324dd2be32fd47a168d402 P 36f86b86532e42758e1f5249c186f526 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "36f86b86532e42758e1f5249c186f526" member_type: VOTER last_known_addr { host: "127.6.234.195" port: 37197 } } peers { permanent_uuid: "47814c4c6447409dbab77568f6434cbd" member_type: VOTER last_known_addr { host: "127.6.234.193" port: 40543 } } peers { permanent_uuid: "170e23b744f94a62854cb7cd24ac7506" member_type: VOTER last_known_addr { host: "127.6.234.194" port: 44189 } }
I20260620 11:42:35.148484 9318 leader_election.cc:288] T fa7dc55b71324dd2be32fd47a168d402 P 36f86b86532e42758e1f5249c186f526 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 47814c4c6447409dbab77568f6434cbd (127.6.234.193:40543), 170e23b744f94a62854cb7cd24ac7506 (127.6.234.194:44189)
I20260620 11:42:35.151780 9183 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 11:42:35.148864 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.234.193:40543 (local address 127.6.234.195:59161)
0620 11:42:35.148925 (+ 61us) negotiation.cc:107] Waiting for socket to connect
0620 11:42:35.148935 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0620 11:42:35.148979 (+ 44us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 11:42:35.149313 (+ 334us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 11:42:35.149314 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 11:42:35.149316 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 11:42:35.149396 (+ 80us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 11:42:35.149399 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 11:42:35.150158 (+ 759us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 11:42:35.150181 (+ 23us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 11:42:35.150965 (+ 784us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 11:42:35.150967 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 11:42:35.151494 (+ 527us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 11:42:35.151496 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 11:42:35.151582 (+ 86us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 11:42:35.151586 (+ 4us) client_negotiation.cc:769] Sending connection context
0620 11:42:35.151643 (+ 57us) client_negotiation.cc:240] Negotiation successful
0620 11:42:35.151697 (+ 54us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":40,"mutex_wait_us":3}
I20260620 11:42:35.151775 9309 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 11:42:35.149091 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.6.234.195:59161 (local address 127.6.234.193:40543)
0620 11:42:35.149147 (+ 56us) server_negotiation.cc:206] Beginning negotiation
0620 11:42:35.149150 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0620 11:42:35.149162 (+ 12us) server_negotiation.cc:407] Connection header received
0620 11:42:35.149194 (+ 32us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 11:42:35.149195 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 11:42:35.149206 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 11:42:35.149281 (+ 75us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 11:42:35.149488 (+ 207us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 11:42:35.150079 (+ 591us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 11:42:35.151058 (+ 979us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 11:42:35.151417 (+ 359us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 11:42:35.151479 (+ 62us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 11:42:35.151486 (+ 7us) server_negotiation.cc:1035] Waiting for connection context
0620 11:42:35.151657 (+ 171us) server_negotiation.cc:299] Negotiation successful
0620 11:42:35.151704 (+ 47us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":27}
I20260620 11:42:35.151823 9308 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 11:42:35.149071 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.6.234.195:47569 (local address 127.6.234.194:44189)
0620 11:42:35.149136 (+ 65us) server_negotiation.cc:206] Beginning negotiation
0620 11:42:35.149140 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 11:42:35.149295 (+ 155us) server_negotiation.cc:407] Connection header received
0620 11:42:35.149326 (+ 31us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 11:42:35.149327 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 11:42:35.149336 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 11:42:35.149400 (+ 64us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 11:42:35.149581 (+ 181us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 11:42:35.150131 (+ 550us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 11:42:35.151102 (+ 971us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 11:42:35.151443 (+ 341us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 11:42:35.151539 (+ 96us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 11:42:35.151556 (+ 17us) server_negotiation.cc:1035] Waiting for connection context
0620 11:42:35.151708 (+ 152us) server_negotiation.cc:299] Negotiation successful
0620 11:42:35.151752 (+ 44us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":42}
I20260620 11:42:35.151875 9319 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 11:42:35.149007 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.234.194:44189 (local address 127.6.234.195:47569)
0620 11:42:35.149218 (+ 211us) negotiation.cc:107] Waiting for socket to connect
0620 11:42:35.149227 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0620 11:42:35.149281 (+ 54us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 11:42:35.149411 (+ 130us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 11:42:35.149411 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 11:42:35.149413 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 11:42:35.149501 (+ 88us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 11:42:35.149504 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 11:42:35.150237 (+ 733us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 11:42:35.150238 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 11:42:35.150978 (+ 740us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 11:42:35.150979 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 11:42:35.151551 (+ 572us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 11:42:35.151552 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 11:42:35.151611 (+ 59us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 11:42:35.151632 (+ 21us) client_negotiation.cc:769] Sending connection context
0620 11:42:35.151690 (+ 58us) client_negotiation.cc:240] Negotiation successful
0620 11:42:35.151724 (+ 34us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":194,"thread_start_us":121,"threads_started":1}
I20260620 11:42:35.152158 8964 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fa7dc55b71324dd2be32fd47a168d402" candidate_uuid: "36f86b86532e42758e1f5249c186f526" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "47814c4c6447409dbab77568f6434cbd" is_pre_election: true
I20260620 11:42:35.152333 8964 raft_consensus.cc:2454] T fa7dc55b71324dd2be32fd47a168d402 P 47814c4c6447409dbab77568f6434cbd [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 36f86b86532e42758e1f5249c186f526 in term 0.
I20260620 11:42:35.152415 9104 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fa7dc55b71324dd2be32fd47a168d402" candidate_uuid: "36f86b86532e42758e1f5249c186f526" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "170e23b744f94a62854cb7cd24ac7506" is_pre_election: true
I20260620 11:42:35.152572 9104 raft_consensus.cc:2454] T fa7dc55b71324dd2be32fd47a168d402 P 170e23b744f94a62854cb7cd24ac7506 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 36f86b86532e42758e1f5249c186f526 in term 0.
I20260620 11:42:35.152623 9175 leader_election.cc:302] T fa7dc55b71324dd2be32fd47a168d402 P 36f86b86532e42758e1f5249c186f526 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 36f86b86532e42758e1f5249c186f526, 47814c4c6447409dbab77568f6434cbd; no voters:
I20260620 11:42:35.152763 9318 raft_consensus.cc:2781] T fa7dc55b71324dd2be32fd47a168d402 P 36f86b86532e42758e1f5249c186f526 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260620 11:42:35.152818 9318 raft_consensus.cc:482] T fa7dc55b71324dd2be32fd47a168d402 P 36f86b86532e42758e1f5249c186f526 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260620 11:42:35.152841 9318 raft_consensus.cc:3037] T fa7dc55b71324dd2be32fd47a168d402 P 36f86b86532e42758e1f5249c186f526 [term 0 FOLLOWER]: Advancing to term 1
I20260620 11:42:35.153564 9318 raft_consensus.cc:504] T fa7dc55b71324dd2be32fd47a168d402 P 36f86b86532e42758e1f5249c186f526 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "36f86b86532e42758e1f5249c186f526" member_type: VOTER last_known_addr { host: "127.6.234.195" port: 37197 } } peers { permanent_uuid: "47814c4c6447409dbab77568f6434cbd" member_type: VOTER last_known_addr { host: "127.6.234.193" port: 40543 } } peers { permanent_uuid: "170e23b744f94a62854cb7cd24ac7506" member_type: VOTER last_known_addr { host: "127.6.234.194" port: 44189 } }
I20260620 11:42:35.153709 9318 leader_election.cc:288] T fa7dc55b71324dd2be32fd47a168d402 P 36f86b86532e42758e1f5249c186f526 [CANDIDATE]: Term 1 election: Requested vote from peers 47814c4c6447409dbab77568f6434cbd (127.6.234.193:40543), 170e23b744f94a62854cb7cd24ac7506 (127.6.234.194:44189)
I20260620 11:42:35.153887 9104 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fa7dc55b71324dd2be32fd47a168d402" candidate_uuid: "36f86b86532e42758e1f5249c186f526" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "170e23b744f94a62854cb7cd24ac7506"
I20260620 11:42:35.153939 8964 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fa7dc55b71324dd2be32fd47a168d402" candidate_uuid: "36f86b86532e42758e1f5249c186f526" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "47814c4c6447409dbab77568f6434cbd"
I20260620 11:42:35.153980 9104 raft_consensus.cc:3037] T fa7dc55b71324dd2be32fd47a168d402 P 170e23b744f94a62854cb7cd24ac7506 [term 0 FOLLOWER]: Advancing to term 1
I20260620 11:42:35.154067 8964 raft_consensus.cc:3037] T fa7dc55b71324dd2be32fd47a168d402 P 47814c4c6447409dbab77568f6434cbd [term 0 FOLLOWER]: Advancing to term 1
I20260620 11:42:35.154695 9104 raft_consensus.cc:2454] T fa7dc55b71324dd2be32fd47a168d402 P 170e23b744f94a62854cb7cd24ac7506 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 36f86b86532e42758e1f5249c186f526 in term 1.
I20260620 11:42:35.154901 8964 raft_consensus.cc:2454] T fa7dc55b71324dd2be32fd47a168d402 P 47814c4c6447409dbab77568f6434cbd [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 36f86b86532e42758e1f5249c186f526 in term 1.
I20260620 11:42:35.155246 9177 leader_election.cc:302] T fa7dc55b71324dd2be32fd47a168d402 P 36f86b86532e42758e1f5249c186f526 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 170e23b744f94a62854cb7cd24ac7506, 36f86b86532e42758e1f5249c186f526; no voters:
I20260620 11:42:35.155361 9318 raft_consensus.cc:2781] T fa7dc55b71324dd2be32fd47a168d402 P 36f86b86532e42758e1f5249c186f526 [term 1 FOLLOWER]: Leader election won for term 1
I20260620 11:42:35.155616 9318 raft_consensus.cc:686] T fa7dc55b71324dd2be32fd47a168d402 P 36f86b86532e42758e1f5249c186f526 [term 1 LEADER]: Becoming Leader. State: Replica: 36f86b86532e42758e1f5249c186f526, State: Running, Role: LEADER
I20260620 11:42:35.155731 9318 consensus_queue.cc:237] T fa7dc55b71324dd2be32fd47a168d402 P 36f86b86532e42758e1f5249c186f526 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "36f86b86532e42758e1f5249c186f526" member_type: VOTER last_known_addr { host: "127.6.234.195" port: 37197 } } peers { permanent_uuid: "47814c4c6447409dbab77568f6434cbd" member_type: VOTER last_known_addr { host: "127.6.234.193" port: 40543 } } peers { permanent_uuid: "170e23b744f94a62854cb7cd24ac7506" member_type: VOTER last_known_addr { host: "127.6.234.194" port: 44189 } }
I20260620 11:42:35.156587 8807 catalog_manager.cc:5495] T fa7dc55b71324dd2be32fd47a168d402 P 36f86b86532e42758e1f5249c186f526 reported cstate change: term changed from 0 to 1, leader changed from <none> to 36f86b86532e42758e1f5249c186f526 (127.6.234.195). New cstate: current_term: 1 leader_uuid: "36f86b86532e42758e1f5249c186f526" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "36f86b86532e42758e1f5249c186f526" member_type: VOTER last_known_addr { host: "127.6.234.195" port: 37197 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "47814c4c6447409dbab77568f6434cbd" member_type: VOTER last_known_addr { host: "127.6.234.193" port: 40543 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "170e23b744f94a62854cb7cd24ac7506" member_type: VOTER last_known_addr { host: "127.6.234.194" port: 44189 } health_report { overall_health: UNKNOWN } } }
Jun 20 11:42:35 dist-test-slave-cjc6 krb5kdc[8762](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781955754, etypes {rep=17 tkt=17 ses=17}, kudu/127.6.234.254@KRBTEST.COM for kudu/127.6.234.254@KRBTEST.COM
I20260620 11:42:35.198309 8863 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 11:42:35.190514 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47682 (local address 127.6.234.254:36185)
0620 11:42:35.190626 (+ 112us) server_negotiation.cc:206] Beginning negotiation
0620 11:42:35.190629 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0620 11:42:35.190725 (+ 96us) server_negotiation.cc:407] Connection header received
0620 11:42:35.191007 (+ 282us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 11:42:35.191008 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 11:42:35.191020 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 11:42:35.191073 (+ 53us) server_negotiation.cc:226] Negotiated authn=SASL
0620 11:42:35.191620 (+ 547us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 11:42:35.192235 (+ 615us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 11:42:35.193152 (+ 917us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 11:42:35.193347 (+ 195us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 11:42:35.193417 (+ 70us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 11:42:35.195574 (+ 2157us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 11:42:35.195575 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 11:42:35.195576 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 11:42:35.195590 (+ 14us) server_negotiation.cc:922] calling sasl_server_start()
0620 11:42:35.197035 (+ 1445us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 11:42:35.197440 (+ 405us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 11:42:35.197441 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 11:42:35.197441 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 11:42:35.197503 (+ 62us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 11:42:35.197724 (+ 221us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 11:42:35.197725 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 11:42:35.197725 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 11:42:35.197835 (+ 110us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 11:42:35.197911 (+ 76us) server_negotiation.cc:1035] Waiting for connection context
0620 11:42:35.198075 (+ 164us) server_negotiation.cc:299] Negotiation successful
0620 11:42:35.198193 (+ 118us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":73}
I20260620 11:42:35.198374 9327 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 11:42:35.190434 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.234.254:36185 (local address 127.0.0.1:47682)
0620 11:42:35.190642 (+ 208us) negotiation.cc:107] Waiting for socket to connect
0620 11:42:35.190652 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0620 11:42:35.190741 (+ 89us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 11:42:35.191140 (+ 399us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 11:42:35.191141 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 11:42:35.191460 (+ 319us) client_negotiation.cc:189] Negotiated authn=SASL
0620 11:42:35.191519 (+ 59us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 11:42:35.191522 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 11:42:35.192337 (+ 815us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 11:42:35.192338 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 11:42:35.193074 (+ 736us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 11:42:35.193077 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 11:42:35.193428 (+ 351us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 11:42:35.193429 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 11:42:35.193475 (+ 46us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 11:42:35.194080 (+ 605us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 11:42:35.194085 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0620 11:42:35.195471 (+ 1386us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 11:42:35.197133 (+ 1662us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 11:42:35.197142 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 11:42:35.197145 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0620 11:42:35.197350 (+ 205us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 11:42:35.197570 (+ 220us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 11:42:35.197571 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 11:42:35.197571 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0620 11:42:35.197634 (+ 63us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 11:42:35.197902 (+ 268us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 11:42:35.197910 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 11:42:35.197959 (+ 49us) client_negotiation.cc:769] Sending connection context
0620 11:42:35.198052 (+ 93us) client_negotiation.cc:240] Negotiation successful
0620 11:42:35.198248 (+ 196us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":143,"mutex_wait_us":90,"thread_start_us":66,"threads_started":1}
I20260620 11:42:35.200956 8807 catalog_manager.cc:1908] Servicing CreateTable request from {username='kudu', principal='kudu/127.6.234.254@KRBTEST.COM'} at 127.0.0.1:47682:
name: "kudu_system.kudu_transactions"
schema {
columns {
name: "txn_id"
type: INT64
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "entry_type"
type: INT8
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "identifier"
type: STRING
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "metadata"
type: STRING
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
rows: "<redacted>""\006\001\000\000\000\000\000\000\000\000\007\001@B\017\000\000\000\000\000"
indirect_data: "<redacted>"""
}
partition_schema {
range_schema {
columns {
name: "txn_id"
}
}
}
table_type: TXN_STATUS_TABLE
W20260620 11:42:35.201268 8807 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table kudu_system.kudu_transactions in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260620 11:42:35.202876 9084 tablet_service.cc:1467] Processing CreateTablet for tablet 1c0c4855fd664eb985f8a1193c7cbcc9 (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=9706597f79974adfa798269cc6275677]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260620 11:42:35.202888 8944 tablet_service.cc:1467] Processing CreateTablet for tablet 1c0c4855fd664eb985f8a1193c7cbcc9 (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=9706597f79974adfa798269cc6275677]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260620 11:42:35.202981 9224 tablet_service.cc:1467] Processing CreateTablet for tablet 1c0c4855fd664eb985f8a1193c7cbcc9 (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=9706597f79974adfa798269cc6275677]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260620 11:42:35.203089 9084 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 1c0c4855fd664eb985f8a1193c7cbcc9. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 11:42:35.203087 8944 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 1c0c4855fd664eb985f8a1193c7cbcc9. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 11:42:35.203128 9224 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 1c0c4855fd664eb985f8a1193c7cbcc9. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 11:42:35.204738 9310 tablet_bootstrap.cc:492] T 1c0c4855fd664eb985f8a1193c7cbcc9 P 47814c4c6447409dbab77568f6434cbd: Bootstrap starting.
I20260620 11:42:35.204732 9312 tablet_bootstrap.cc:492] T 1c0c4855fd664eb985f8a1193c7cbcc9 P 170e23b744f94a62854cb7cd24ac7506: Bootstrap starting.
I20260620 11:42:35.205191 9310 tablet_bootstrap.cc:654] T 1c0c4855fd664eb985f8a1193c7cbcc9 P 47814c4c6447409dbab77568f6434cbd: Neither blocks nor log segments found. Creating new log.
I20260620 11:42:35.205192 9312 tablet_bootstrap.cc:654] T 1c0c4855fd664eb985f8a1193c7cbcc9 P 170e23b744f94a62854cb7cd24ac7506: Neither blocks nor log segments found. Creating new log.
I20260620 11:42:35.205523 9311 tablet_bootstrap.cc:492] T 1c0c4855fd664eb985f8a1193c7cbcc9 P 36f86b86532e42758e1f5249c186f526: Bootstrap starting.
I20260620 11:42:35.205895 9310 tablet_bootstrap.cc:492] T 1c0c4855fd664eb985f8a1193c7cbcc9 P 47814c4c6447409dbab77568f6434cbd: No bootstrap required, opened a new log
I20260620 11:42:35.205955 9312 tablet_bootstrap.cc:492] T 1c0c4855fd664eb985f8a1193c7cbcc9 P 170e23b744f94a62854cb7cd24ac7506: No bootstrap required, opened a new log
I20260620 11:42:35.205951 9310 ts_tablet_manager.cc:1397] T 1c0c4855fd664eb985f8a1193c7cbcc9 P 47814c4c6447409dbab77568f6434cbd: Time spent bootstrapping tablet: real 0.001s user 0.001s sys 0.000s
I20260620 11:42:35.206000 9312 ts_tablet_manager.cc:1397] T 1c0c4855fd664eb985f8a1193c7cbcc9 P 170e23b744f94a62854cb7cd24ac7506: Time spent bootstrapping tablet: real 0.001s user 0.001s sys 0.000s
I20260620 11:42:35.206028 9311 tablet_bootstrap.cc:654] T 1c0c4855fd664eb985f8a1193c7cbcc9 P 36f86b86532e42758e1f5249c186f526: Neither blocks nor log segments found. Creating new log.
I20260620 11:42:35.206126 9310 raft_consensus.cc:348] T 1c0c4855fd664eb985f8a1193c7cbcc9 P 47814c4c6447409dbab77568f6434cbd [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "47814c4c6447409dbab77568f6434cbd" member_type: VOTER last_known_addr { host: "127.6.234.193" port: 40543 } } peers { permanent_uuid: "170e23b744f94a62854cb7cd24ac7506" member_type: VOTER last_known_addr { host: "127.6.234.194" port: 44189 } } peers { permanent_uuid: "36f86b86532e42758e1f5249c186f526" member_type: VOTER last_known_addr { host: "127.6.234.195" port: 37197 } }
I20260620 11:42:35.206199 9310 raft_consensus.cc:374] T 1c0c4855fd664eb985f8a1193c7cbcc9 P 47814c4c6447409dbab77568f6434cbd [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 11:42:35.206161 9312 raft_consensus.cc:348] T 1c0c4855fd664eb985f8a1193c7cbcc9 P 170e23b744f94a62854cb7cd24ac7506 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "47814c4c6447409dbab77568f6434cbd" member_type: VOTER last_known_addr { host: "127.6.234.193" port: 40543 } } peers { permanent_uuid: "170e23b744f94a62854cb7cd24ac7506" member_type: VOTER last_known_addr { host: "127.6.234.194" port: 44189 } } peers { permanent_uuid: "36f86b86532e42758e1f5249c186f526" member_type: VOTER last_known_addr { host: "127.6.234.195" port: 37197 } }
I20260620 11:42:35.206223 9310 raft_consensus.cc:729] T 1c0c4855fd664eb985f8a1193c7cbcc9 P 47814c4c6447409dbab77568f6434cbd [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 47814c4c6447409dbab77568f6434cbd, State: Initialized, Role: FOLLOWER
I20260620 11:42:35.206228 9312 raft_consensus.cc:374] T 1c0c4855fd664eb985f8a1193c7cbcc9 P 170e23b744f94a62854cb7cd24ac7506 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 11:42:35.206254 9312 raft_consensus.cc:729] T 1c0c4855fd664eb985f8a1193c7cbcc9 P 170e23b744f94a62854cb7cd24ac7506 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 170e23b744f94a62854cb7cd24ac7506, State: Initialized, Role: FOLLOWER
I20260620 11:42:35.206277 9310 consensus_queue.cc:260] T 1c0c4855fd664eb985f8a1193c7cbcc9 P 47814c4c6447409dbab77568f6434cbd [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "47814c4c6447409dbab77568f6434cbd" member_type: VOTER last_known_addr { host: "127.6.234.193" port: 40543 } } peers { permanent_uuid: "170e23b744f94a62854cb7cd24ac7506" member_type: VOTER last_known_addr { host: "127.6.234.194" port: 44189 } } peers { permanent_uuid: "36f86b86532e42758e1f5249c186f526" member_type: VOTER last_known_addr { host: "127.6.234.195" port: 37197 } }
I20260620 11:42:35.206305 9312 consensus_queue.cc:260] T 1c0c4855fd664eb985f8a1193c7cbcc9 P 170e23b744f94a62854cb7cd24ac7506 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "47814c4c6447409dbab77568f6434cbd" member_type: VOTER last_known_addr { host: "127.6.234.193" port: 40543 } } peers { permanent_uuid: "170e23b744f94a62854cb7cd24ac7506" member_type: VOTER last_known_addr { host: "127.6.234.194" port: 44189 } } peers { permanent_uuid: "36f86b86532e42758e1f5249c186f526" member_type: VOTER last_known_addr { host: "127.6.234.195" port: 37197 } }
I20260620 11:42:35.206387 9310 ts_tablet_manager.cc:1428] T 1c0c4855fd664eb985f8a1193c7cbcc9 P 47814c4c6447409dbab77568f6434cbd: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
I20260620 11:42:35.206413 9312 ts_tablet_manager.cc:1428] T 1c0c4855fd664eb985f8a1193c7cbcc9 P 170e23b744f94a62854cb7cd24ac7506: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
I20260620 11:42:35.206456 9317 tablet_replica.cc:439] T 1c0c4855fd664eb985f8a1193c7cbcc9 P 170e23b744f94a62854cb7cd24ac7506: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "47814c4c6447409dbab77568f6434cbd" member_type: VOTER last_known_addr { host: "127.6.234.193" port: 40543 } } peers { permanent_uuid: "170e23b744f94a62854cb7cd24ac7506" member_type: VOTER last_known_addr { host: "127.6.234.194" port: 44189 } } peers { permanent_uuid: "36f86b86532e42758e1f5249c186f526" member_type: VOTER last_known_addr { host: "127.6.234.195" port: 37197 } } }
I20260620 11:42:35.206563 9317 tablet_replica.cc:442] T 1c0c4855fd664eb985f8a1193c7cbcc9 P 170e23b744f94a62854cb7cd24ac7506: This TxnStatusTablet replica's current role is: FOLLOWER
I20260620 11:42:35.206574 9316 tablet_replica.cc:439] T 1c0c4855fd664eb985f8a1193c7cbcc9 P 47814c4c6447409dbab77568f6434cbd: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "47814c4c6447409dbab77568f6434cbd" member_type: VOTER last_known_addr { host: "127.6.234.193" port: 40543 } } peers { permanent_uuid: "170e23b744f94a62854cb7cd24ac7506" member_type: VOTER last_known_addr { host: "127.6.234.194" port: 44189 } } peers { permanent_uuid: "36f86b86532e42758e1f5249c186f526" member_type: VOTER last_known_addr { host: "127.6.234.195" port: 37197 } } }
I20260620 11:42:35.206657 9316 tablet_replica.cc:442] T 1c0c4855fd664eb985f8a1193c7cbcc9 P 47814c4c6447409dbab77568f6434cbd: This TxnStatusTablet replica's current role is: FOLLOWER
I20260620 11:42:35.207401 9311 tablet_bootstrap.cc:492] T 1c0c4855fd664eb985f8a1193c7cbcc9 P 36f86b86532e42758e1f5249c186f526: No bootstrap required, opened a new log
I20260620 11:42:35.207446 9311 ts_tablet_manager.cc:1397] T 1c0c4855fd664eb985f8a1193c7cbcc9 P 36f86b86532e42758e1f5249c186f526: Time spent bootstrapping tablet: real 0.002s user 0.000s sys 0.001s
I20260620 11:42:35.207610 9311 raft_consensus.cc:348] T 1c0c4855fd664eb985f8a1193c7cbcc9 P 36f86b86532e42758e1f5249c186f526 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "47814c4c6447409dbab77568f6434cbd" member_type: VOTER last_known_addr { host: "127.6.234.193" port: 40543 } } peers { permanent_uuid: "170e23b744f94a62854cb7cd24ac7506" member_type: VOTER last_known_addr { host: "127.6.234.194" port: 44189 } } peers { permanent_uuid: "36f86b86532e42758e1f5249c186f526" member_type: VOTER last_known_addr { host: "127.6.234.195" port: 37197 } }
I20260620 11:42:35.207675 9311 raft_consensus.cc:374] T 1c0c4855fd664eb985f8a1193c7cbcc9 P 36f86b86532e42758e1f5249c186f526 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 11:42:35.207688 9311 raft_consensus.cc:729] T 1c0c4855fd664eb985f8a1193c7cbcc9 P 36f86b86532e42758e1f5249c186f526 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 36f86b86532e42758e1f5249c186f526, State: Initialized, Role: FOLLOWER
I20260620 11:42:35.207746 9311 consensus_queue.cc:260] T 1c0c4855fd664eb985f8a1193c7cbcc9 P 36f86b86532e42758e1f5249c186f526 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "47814c4c6447409dbab77568f6434cbd" member_type: VOTER last_known_addr { host: "127.6.234.193" port: 40543 } } peers { permanent_uuid: "170e23b744f94a62854cb7cd24ac7506" member_type: VOTER last_known_addr { host: "127.6.234.194" port: 44189 } } peers { permanent_uuid: "36f86b86532e42758e1f5249c186f526" member_type: VOTER last_known_addr { host: "127.6.234.195" port: 37197 } }
I20260620 11:42:35.207854 9311 ts_tablet_manager.cc:1428] T 1c0c4855fd664eb985f8a1193c7cbcc9 P 36f86b86532e42758e1f5249c186f526: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
I20260620 11:42:35.207983 9318 tablet_replica.cc:439] T 1c0c4855fd664eb985f8a1193c7cbcc9 P 36f86b86532e42758e1f5249c186f526: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "47814c4c6447409dbab77568f6434cbd" member_type: VOTER last_known_addr { host: "127.6.234.193" port: 40543 } } peers { permanent_uuid: "170e23b744f94a62854cb7cd24ac7506" member_type: VOTER last_known_addr { host: "127.6.234.194" port: 44189 } } peers { permanent_uuid: "36f86b86532e42758e1f5249c186f526" member_type: VOTER last_known_addr { host: "127.6.234.195" port: 37197 } } }
I20260620 11:42:35.208078 9318 tablet_replica.cc:442] T 1c0c4855fd664eb985f8a1193c7cbcc9 P 36f86b86532e42758e1f5249c186f526: This TxnStatusTablet replica's current role is: FOLLOWER
I20260620 11:42:35.253827 9318 raft_consensus.cc:482] T 1c0c4855fd664eb985f8a1193c7cbcc9 P 36f86b86532e42758e1f5249c186f526 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260620 11:42:35.253952 9318 raft_consensus.cc:504] T 1c0c4855fd664eb985f8a1193c7cbcc9 P 36f86b86532e42758e1f5249c186f526 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "47814c4c6447409dbab77568f6434cbd" member_type: VOTER last_known_addr { host: "127.6.234.193" port: 40543 } } peers { permanent_uuid: "170e23b744f94a62854cb7cd24ac7506" member_type: VOTER last_known_addr { host: "127.6.234.194" port: 44189 } } peers { permanent_uuid: "36f86b86532e42758e1f5249c186f526" member_type: VOTER last_known_addr { host: "127.6.234.195" port: 37197 } }
I20260620 11:42:35.254155 9318 leader_election.cc:288] T 1c0c4855fd664eb985f8a1193c7cbcc9 P 36f86b86532e42758e1f5249c186f526 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 47814c4c6447409dbab77568f6434cbd (127.6.234.193:40543), 170e23b744f94a62854cb7cd24ac7506 (127.6.234.194:44189)
I20260620 11:42:35.254434 8964 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1c0c4855fd664eb985f8a1193c7cbcc9" candidate_uuid: "36f86b86532e42758e1f5249c186f526" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "47814c4c6447409dbab77568f6434cbd" is_pre_election: true
I20260620 11:42:35.254470 9104 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1c0c4855fd664eb985f8a1193c7cbcc9" candidate_uuid: "36f86b86532e42758e1f5249c186f526" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "170e23b744f94a62854cb7cd24ac7506" is_pre_election: true
I20260620 11:42:35.254546 8964 raft_consensus.cc:2454] T 1c0c4855fd664eb985f8a1193c7cbcc9 P 47814c4c6447409dbab77568f6434cbd [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 36f86b86532e42758e1f5249c186f526 in term 0.
I20260620 11:42:35.254585 9104 raft_consensus.cc:2454] T 1c0c4855fd664eb985f8a1193c7cbcc9 P 170e23b744f94a62854cb7cd24ac7506 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 36f86b86532e42758e1f5249c186f526 in term 0.
I20260620 11:42:35.254698 9175 leader_election.cc:302] T 1c0c4855fd664eb985f8a1193c7cbcc9 P 36f86b86532e42758e1f5249c186f526 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 36f86b86532e42758e1f5249c186f526, 47814c4c6447409dbab77568f6434cbd; no voters:
I20260620 11:42:35.254807 9318 raft_consensus.cc:2781] T 1c0c4855fd664eb985f8a1193c7cbcc9 P 36f86b86532e42758e1f5249c186f526 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260620 11:42:35.254863 9318 raft_consensus.cc:482] T 1c0c4855fd664eb985f8a1193c7cbcc9 P 36f86b86532e42758e1f5249c186f526 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260620 11:42:35.254900 9318 raft_consensus.cc:3037] T 1c0c4855fd664eb985f8a1193c7cbcc9 P 36f86b86532e42758e1f5249c186f526 [term 0 FOLLOWER]: Advancing to term 1
I20260620 11:42:35.255652 9318 raft_consensus.cc:504] T 1c0c4855fd664eb985f8a1193c7cbcc9 P 36f86b86532e42758e1f5249c186f526 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "47814c4c6447409dbab77568f6434cbd" member_type: VOTER last_known_addr { host: "127.6.234.193" port: 40543 } } peers { permanent_uuid: "170e23b744f94a62854cb7cd24ac7506" member_type: VOTER last_known_addr { host: "127.6.234.194" port: 44189 } } peers { permanent_uuid: "36f86b86532e42758e1f5249c186f526" member_type: VOTER last_known_addr { host: "127.6.234.195" port: 37197 } }
I20260620 11:42:35.255805 9318 leader_election.cc:288] T 1c0c4855fd664eb985f8a1193c7cbcc9 P 36f86b86532e42758e1f5249c186f526 [CANDIDATE]: Term 1 election: Requested vote from peers 47814c4c6447409dbab77568f6434cbd (127.6.234.193:40543), 170e23b744f94a62854cb7cd24ac7506 (127.6.234.194:44189)
I20260620 11:42:35.255967 9104 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1c0c4855fd664eb985f8a1193c7cbcc9" candidate_uuid: "36f86b86532e42758e1f5249c186f526" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "170e23b744f94a62854cb7cd24ac7506"
I20260620 11:42:35.256074 9104 raft_consensus.cc:3037] T 1c0c4855fd664eb985f8a1193c7cbcc9 P 170e23b744f94a62854cb7cd24ac7506 [term 0 FOLLOWER]: Advancing to term 1
I20260620 11:42:35.256042 8964 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1c0c4855fd664eb985f8a1193c7cbcc9" candidate_uuid: "36f86b86532e42758e1f5249c186f526" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "47814c4c6447409dbab77568f6434cbd"
I20260620 11:42:35.256114 8964 raft_consensus.cc:3037] T 1c0c4855fd664eb985f8a1193c7cbcc9 P 47814c4c6447409dbab77568f6434cbd [term 0 FOLLOWER]: Advancing to term 1
I20260620 11:42:35.256820 9104 raft_consensus.cc:2454] T 1c0c4855fd664eb985f8a1193c7cbcc9 P 170e23b744f94a62854cb7cd24ac7506 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 36f86b86532e42758e1f5249c186f526 in term 1.
I20260620 11:42:35.256820 8964 raft_consensus.cc:2454] T 1c0c4855fd664eb985f8a1193c7cbcc9 P 47814c4c6447409dbab77568f6434cbd [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 36f86b86532e42758e1f5249c186f526 in term 1.
I20260620 11:42:35.256991 9175 leader_election.cc:302] T 1c0c4855fd664eb985f8a1193c7cbcc9 P 36f86b86532e42758e1f5249c186f526 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 36f86b86532e42758e1f5249c186f526, 47814c4c6447409dbab77568f6434cbd; no voters:
I20260620 11:42:35.257085 9318 raft_consensus.cc:2781] T 1c0c4855fd664eb985f8a1193c7cbcc9 P 36f86b86532e42758e1f5249c186f526 [term 1 FOLLOWER]: Leader election won for term 1
I20260620 11:42:35.257138 9318 raft_consensus.cc:686] T 1c0c4855fd664eb985f8a1193c7cbcc9 P 36f86b86532e42758e1f5249c186f526 [term 1 LEADER]: Becoming Leader. State: Replica: 36f86b86532e42758e1f5249c186f526, State: Running, Role: LEADER
I20260620 11:42:35.257181 9318 consensus_queue.cc:237] T 1c0c4855fd664eb985f8a1193c7cbcc9 P 36f86b86532e42758e1f5249c186f526 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "47814c4c6447409dbab77568f6434cbd" member_type: VOTER last_known_addr { host: "127.6.234.193" port: 40543 } } peers { permanent_uuid: "170e23b744f94a62854cb7cd24ac7506" member_type: VOTER last_known_addr { host: "127.6.234.194" port: 44189 } } peers { permanent_uuid: "36f86b86532e42758e1f5249c186f526" member_type: VOTER last_known_addr { host: "127.6.234.195" port: 37197 } }
I20260620 11:42:35.257495 9320 tablet_replica.cc:439] T 1c0c4855fd664eb985f8a1193c7cbcc9 P 36f86b86532e42758e1f5249c186f526: TxnStatusTablet state changed. Reason: New leader 36f86b86532e42758e1f5249c186f526. Latest consensus state: current_term: 1 leader_uuid: "36f86b86532e42758e1f5249c186f526" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "47814c4c6447409dbab77568f6434cbd" member_type: VOTER last_known_addr { host: "127.6.234.193" port: 40543 } } peers { permanent_uuid: "170e23b744f94a62854cb7cd24ac7506" member_type: VOTER last_known_addr { host: "127.6.234.194" port: 44189 } } peers { permanent_uuid: "36f86b86532e42758e1f5249c186f526" member_type: VOTER last_known_addr { host: "127.6.234.195" port: 37197 } } }
I20260620 11:42:35.257596 9320 tablet_replica.cc:442] T 1c0c4855fd664eb985f8a1193c7cbcc9 P 36f86b86532e42758e1f5249c186f526: This TxnStatusTablet replica's current role is: LEADER
I20260620 11:42:35.257859 9332 txn_status_manager.cc:873] Waiting until node catch up with all replicated operations in previous term...
I20260620 11:42:35.257861 8807 catalog_manager.cc:5495] T 1c0c4855fd664eb985f8a1193c7cbcc9 P 36f86b86532e42758e1f5249c186f526 reported cstate change: term changed from 0 to 1, leader changed from <none> to 36f86b86532e42758e1f5249c186f526 (127.6.234.195). New cstate: current_term: 1 leader_uuid: "36f86b86532e42758e1f5249c186f526" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "47814c4c6447409dbab77568f6434cbd" member_type: VOTER last_known_addr { host: "127.6.234.193" port: 40543 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "170e23b744f94a62854cb7cd24ac7506" member_type: VOTER last_known_addr { host: "127.6.234.194" port: 44189 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "36f86b86532e42758e1f5249c186f526" member_type: VOTER last_known_addr { host: "127.6.234.195" port: 37197 } health_report { overall_health: HEALTHY } } }
I20260620 11:42:35.258011 9332 txn_status_manager.cc:929] Loading transaction status metadata into memory...
I20260620 11:42:35.266460 9307 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 11:42:35.263546 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45112 (local address 127.6.234.195:37197)
0620 11:42:35.263585 (+ 39us) server_negotiation.cc:206] Beginning negotiation
0620 11:42:35.263587 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 11:42:35.263652 (+ 65us) server_negotiation.cc:407] Connection header received
0620 11:42:35.263763 (+ 111us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 11:42:35.263764 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 11:42:35.263778 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 11:42:35.263836 (+ 58us) server_negotiation.cc:226] Negotiated authn=TOKEN
0620 11:42:35.263980 (+ 144us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 11:42:35.264456 (+ 476us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 11:42:35.265572 (+ 1116us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 11:42:35.265838 (+ 266us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 11:42:35.265896 (+ 58us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 11:42:35.266097 (+ 201us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0620 11:42:35.266186 (+ 89us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0620 11:42:35.266244 (+ 58us) server_negotiation.cc:1035] Waiting for connection context
0620 11:42:35.266320 (+ 76us) server_negotiation.cc:299] Negotiation successful
0620 11:42:35.266346 (+ 26us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":18}
I20260620 11:42:35.266458 9327 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 11:42:35.263495 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.234.195:37197 (local address 127.0.0.1:45112)
0620 11:42:35.263599 (+ 104us) negotiation.cc:107] Waiting for socket to connect
0620 11:42:35.263612 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0620 11:42:35.263678 (+ 66us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 11:42:35.263844 (+ 166us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 11:42:35.263845 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 11:42:35.263847 (+ 2us) client_negotiation.cc:189] Negotiated authn=TOKEN
0620 11:42:35.263913 (+ 66us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 11:42:35.263916 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 11:42:35.264543 (+ 627us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 11:42:35.264545 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 11:42:35.265459 (+ 914us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 11:42:35.265462 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 11:42:35.265912 (+ 450us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 11:42:35.265913 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 11:42:35.265984 (+ 71us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 11:42:35.266005 (+ 21us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0620 11:42:35.266249 (+ 244us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0620 11:42:35.266257 (+ 8us) client_negotiation.cc:769] Sending connection context
0620 11:42:35.266312 (+ 55us) client_negotiation.cc:240] Negotiation successful
0620 11:42:35.266366 (+ 54us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":67}
I20260620 11:42:35.268299 9104 raft_consensus.cc:1261] T 1c0c4855fd664eb985f8a1193c7cbcc9 P 170e23b744f94a62854cb7cd24ac7506 [term 1 FOLLOWER]: Refusing update from remote peer 36f86b86532e42758e1f5249c186f526: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 11:42:35.268299 8964 raft_consensus.cc:1261] T 1c0c4855fd664eb985f8a1193c7cbcc9 P 47814c4c6447409dbab77568f6434cbd [term 1 FOLLOWER]: Refusing update from remote peer 36f86b86532e42758e1f5249c186f526: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 11:42:35.268644 9318 consensus_queue.cc:1035] T 1c0c4855fd664eb985f8a1193c7cbcc9 P 36f86b86532e42758e1f5249c186f526 [LEADER]: Connected to new peer: Peer: permanent_uuid: "47814c4c6447409dbab77568f6434cbd" member_type: VOTER last_known_addr { host: "127.6.234.193" port: 40543 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 11:42:35.268793 9320 consensus_queue.cc:1035] T 1c0c4855fd664eb985f8a1193c7cbcc9 P 36f86b86532e42758e1f5249c186f526 [LEADER]: Connected to new peer: Peer: permanent_uuid: "170e23b744f94a62854cb7cd24ac7506" member_type: VOTER last_known_addr { host: "127.6.234.194" port: 44189 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 11:42:35.269982 9316 tablet_replica.cc:439] T 1c0c4855fd664eb985f8a1193c7cbcc9 P 47814c4c6447409dbab77568f6434cbd: TxnStatusTablet state changed. Reason: New leader 36f86b86532e42758e1f5249c186f526. Latest consensus state: current_term: 1 leader_uuid: "36f86b86532e42758e1f5249c186f526" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "47814c4c6447409dbab77568f6434cbd" member_type: VOTER last_known_addr { host: "127.6.234.193" port: 40543 } } peers { permanent_uuid: "170e23b744f94a62854cb7cd24ac7506" member_type: VOTER last_known_addr { host: "127.6.234.194" port: 44189 } } peers { permanent_uuid: "36f86b86532e42758e1f5249c186f526" member_type: VOTER last_known_addr { host: "127.6.234.195" port: 37197 } } }
I20260620 11:42:35.270097 9316 tablet_replica.cc:442] T 1c0c4855fd664eb985f8a1193c7cbcc9 P 47814c4c6447409dbab77568f6434cbd: This TxnStatusTablet replica's current role is: FOLLOWER
I20260620 11:42:35.270641 9317 tablet_replica.cc:439] T 1c0c4855fd664eb985f8a1193c7cbcc9 P 170e23b744f94a62854cb7cd24ac7506: TxnStatusTablet state changed. Reason: New leader 36f86b86532e42758e1f5249c186f526. Latest consensus state: current_term: 1 leader_uuid: "36f86b86532e42758e1f5249c186f526" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "47814c4c6447409dbab77568f6434cbd" member_type: VOTER last_known_addr { host: "127.6.234.193" port: 40543 } } peers { permanent_uuid: "170e23b744f94a62854cb7cd24ac7506" member_type: VOTER last_known_addr { host: "127.6.234.194" port: 44189 } } peers { permanent_uuid: "36f86b86532e42758e1f5249c186f526" member_type: VOTER last_known_addr { host: "127.6.234.195" port: 37197 } } }
I20260620 11:42:35.270731 9317 tablet_replica.cc:442] T 1c0c4855fd664eb985f8a1193c7cbcc9 P 170e23b744f94a62854cb7cd24ac7506: This TxnStatusTablet replica's current role is: FOLLOWER
I20260620 11:42:35.271153 9318 tablet_replica.cc:439] T 1c0c4855fd664eb985f8a1193c7cbcc9 P 36f86b86532e42758e1f5249c186f526: TxnStatusTablet state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "36f86b86532e42758e1f5249c186f526" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "47814c4c6447409dbab77568f6434cbd" member_type: VOTER last_known_addr { host: "127.6.234.193" port: 40543 } } peers { permanent_uuid: "170e23b744f94a62854cb7cd24ac7506" member_type: VOTER last_known_addr { host: "127.6.234.194" port: 44189 } } peers { permanent_uuid: "36f86b86532e42758e1f5249c186f526" member_type: VOTER last_known_addr { host: "127.6.234.195" port: 37197 } } }
I20260620 11:42:35.271241 9318 tablet_replica.cc:442] T 1c0c4855fd664eb985f8a1193c7cbcc9 P 36f86b86532e42758e1f5249c186f526: This TxnStatusTablet replica's current role is: LEADER
I20260620 11:42:35.271804 9320 tablet_replica.cc:439] T 1c0c4855fd664eb985f8a1193c7cbcc9 P 36f86b86532e42758e1f5249c186f526: TxnStatusTablet state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "36f86b86532e42758e1f5249c186f526" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "47814c4c6447409dbab77568f6434cbd" member_type: VOTER last_known_addr { host: "127.6.234.193" port: 40543 } } peers { permanent_uuid: "170e23b744f94a62854cb7cd24ac7506" member_type: VOTER last_known_addr { host: "127.6.234.194" port: 44189 } } peers { permanent_uuid: "36f86b86532e42758e1f5249c186f526" member_type: VOTER last_known_addr { host: "127.6.234.195" port: 37197 } } }
I20260620 11:42:35.271884 9320 tablet_replica.cc:442] T 1c0c4855fd664eb985f8a1193c7cbcc9 P 36f86b86532e42758e1f5249c186f526: This TxnStatusTablet replica's current role is: LEADER
I20260620 11:42:35.272111 9316 tablet_replica.cc:439] T 1c0c4855fd664eb985f8a1193c7cbcc9 P 47814c4c6447409dbab77568f6434cbd: TxnStatusTablet state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "36f86b86532e42758e1f5249c186f526" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "47814c4c6447409dbab77568f6434cbd" member_type: VOTER last_known_addr { host: "127.6.234.193" port: 40543 } } peers { permanent_uuid: "170e23b744f94a62854cb7cd24ac7506" member_type: VOTER last_known_addr { host: "127.6.234.194" port: 44189 } } peers { permanent_uuid: "36f86b86532e42758e1f5249c186f526" member_type: VOTER last_known_addr { host: "127.6.234.195" port: 37197 } } }
I20260620 11:42:35.272195 9316 tablet_replica.cc:442] T 1c0c4855fd664eb985f8a1193c7cbcc9 P 47814c4c6447409dbab77568f6434cbd: This TxnStatusTablet replica's current role is: FOLLOWER
I20260620 11:42:35.272725 9317 tablet_replica.cc:439] T 1c0c4855fd664eb985f8a1193c7cbcc9 P 170e23b744f94a62854cb7cd24ac7506: TxnStatusTablet state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "36f86b86532e42758e1f5249c186f526" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "47814c4c6447409dbab77568f6434cbd" member_type: VOTER last_known_addr { host: "127.6.234.193" port: 40543 } } peers { permanent_uuid: "170e23b744f94a62854cb7cd24ac7506" member_type: VOTER last_known_addr { host: "127.6.234.194" port: 44189 } } peers { permanent_uuid: "36f86b86532e42758e1f5249c186f526" member_type: VOTER last_known_addr { host: "127.6.234.195" port: 37197 } } }
I20260620 11:42:35.272837 9317 tablet_replica.cc:442] T 1c0c4855fd664eb985f8a1193c7cbcc9 P 170e23b744f94a62854cb7cd24ac7506: This TxnStatusTablet replica's current role is: FOLLOWER
I20260620 11:42:35.276969 9307 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 11:42:35.274299 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45120 (local address 127.6.234.195:37197)
0620 11:42:35.274340 (+ 41us) server_negotiation.cc:206] Beginning negotiation
0620 11:42:35.274341 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0620 11:42:35.274352 (+ 11us) server_negotiation.cc:407] Connection header received
0620 11:42:35.274405 (+ 53us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 11:42:35.274405 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 11:42:35.274415 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 11:42:35.274487 (+ 72us) server_negotiation.cc:226] Negotiated authn=TOKEN
0620 11:42:35.274570 (+ 83us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 11:42:35.275299 (+ 729us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 11:42:35.276188 (+ 889us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 11:42:35.276500 (+ 312us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 11:42:35.276572 (+ 72us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 11:42:35.276733 (+ 161us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0620 11:42:35.276768 (+ 35us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0620 11:42:35.276820 (+ 52us) server_negotiation.cc:1035] Waiting for connection context
0620 11:42:35.276866 (+ 46us) server_negotiation.cc:299] Negotiation successful
0620 11:42:35.276887 (+ 21us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":22}
I20260620 11:42:35.282609 9307 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 11:42:35.279763 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.6.234.195:51571 (local address 127.6.234.195:37197)
0620 11:42:35.279797 (+ 34us) server_negotiation.cc:206] Beginning negotiation
0620 11:42:35.279798 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0620 11:42:35.279843 (+ 45us) server_negotiation.cc:407] Connection header received
0620 11:42:35.279915 (+ 72us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 11:42:35.279916 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 11:42:35.279925 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 11:42:35.279971 (+ 46us) server_negotiation.cc:226] Negotiated authn=TOKEN
0620 11:42:35.280422 (+ 451us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 11:42:35.280852 (+ 430us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 11:42:35.281956 (+ 1104us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 11:42:35.282134 (+ 178us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 11:42:35.282186 (+ 52us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 11:42:35.282381 (+ 195us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0620 11:42:35.282406 (+ 25us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0620 11:42:35.282443 (+ 37us) server_negotiation.cc:1035] Waiting for connection context
0620 11:42:35.282523 (+ 80us) server_negotiation.cc:299] Negotiation successful
0620 11:42:35.282537 (+ 14us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":17}
I20260620 11:42:35.282692 9296 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 11:42:35.279709 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.234.195:37197 (local address 127.6.234.195:51571)
0620 11:42:35.279797 (+ 88us) negotiation.cc:107] Waiting for socket to connect
0620 11:42:35.279808 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0620 11:42:35.279860 (+ 52us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 11:42:35.280005 (+ 145us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 11:42:35.280005 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 11:42:35.280007 (+ 2us) client_negotiation.cc:189] Negotiated authn=TOKEN
0620 11:42:35.280351 (+ 344us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 11:42:35.280353 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 11:42:35.280923 (+ 570us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 11:42:35.280924 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 11:42:35.281883 (+ 959us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 11:42:35.281886 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 11:42:35.282201 (+ 315us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 11:42:35.282202 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 11:42:35.282268 (+ 66us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 11:42:35.282284 (+ 16us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0620 11:42:35.282466 (+ 182us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0620 11:42:35.282468 (+ 2us) client_negotiation.cc:769] Sending connection context
0620 11:42:35.282522 (+ 54us) client_negotiation.cc:240] Negotiation successful
0620 11:42:35.282567 (+ 45us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":53}
I20260620 11:42:35.287345 8964 raft_consensus.cc:1261] T fa7dc55b71324dd2be32fd47a168d402 P 47814c4c6447409dbab77568f6434cbd [term 1 FOLLOWER]: Refusing update from remote peer 36f86b86532e42758e1f5249c186f526: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 11:42:35.287381 9104 raft_consensus.cc:1261] T fa7dc55b71324dd2be32fd47a168d402 P 170e23b744f94a62854cb7cd24ac7506 [term 1 FOLLOWER]: Refusing update from remote peer 36f86b86532e42758e1f5249c186f526: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 11:42:35.287590 9346 consensus_queue.cc:1035] T fa7dc55b71324dd2be32fd47a168d402 P 36f86b86532e42758e1f5249c186f526 [LEADER]: Connected to new peer: Peer: permanent_uuid: "47814c4c6447409dbab77568f6434cbd" member_type: VOTER last_known_addr { host: "127.6.234.193" port: 40543 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 11:42:35.287691 9320 consensus_queue.cc:1035] T fa7dc55b71324dd2be32fd47a168d402 P 36f86b86532e42758e1f5249c186f526 [LEADER]: Connected to new peer: Peer: permanent_uuid: "170e23b744f94a62854cb7cd24ac7506" member_type: VOTER last_known_addr { host: "127.6.234.194" port: 44189 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 11:42:35.289199 9336 mvcc.cc:205] Tried to move back new op lower bound from 7298890773655711744 to 7298890773118435328. Current Snapshot: MvccSnapshot[applied={T|T < 7298890773655711744}]
I20260620 11:42:35.290259 9338 mvcc.cc:205] Tried to move back new op lower bound from 7298890773655711744 to 7298890773118435328. Current Snapshot: MvccSnapshot[applied={T|T < 7298890773655711744}]
I20260620 11:42:35.299945 8805 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:47676:
table { table_name: "test-table" } modify_external_catalogs: true
I20260620 11:42:35.300055 8805 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:47676:
table { table_name: "test-table" } modify_external_catalogs: true
I20260620 11:42:35.300772 8805 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 6e89f37d51dc404c95261d264710ef80: Sending DeleteTablet for 3 replicas of tablet fa7dc55b71324dd2be32fd47a168d402
I20260620 11:42:35.301128 9084 tablet_service.cc:1514] Processing DeleteTablet for tablet fa7dc55b71324dd2be32fd47a168d402 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 11:42:35 UTC) from {username='kudu', principal='kudu/127.6.234.254@KRBTEST.COM'} at 127.0.0.1:53692
I20260620 11:42:35.301128 8944 tablet_service.cc:1514] Processing DeleteTablet for tablet fa7dc55b71324dd2be32fd47a168d402 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 11:42:35 UTC) from {username='kudu', principal='kudu/127.6.234.254@KRBTEST.COM'} at 127.0.0.1:58340
I20260620 11:42:35.301375 9350 tablet_replica.cc:330] T fa7dc55b71324dd2be32fd47a168d402 P 47814c4c6447409dbab77568f6434cbd: stopping tablet replica
I20260620 11:42:35.301502 9350 raft_consensus.cc:2229] T fa7dc55b71324dd2be32fd47a168d402 P 47814c4c6447409dbab77568f6434cbd [term 1 FOLLOWER]: Raft consensus shutting down.
I20260620 11:42:35.301448 8805 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:47676:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20260620 11:42:35.301566 9350 raft_consensus.cc:2258] T fa7dc55b71324dd2be32fd47a168d402 P 47814c4c6447409dbab77568f6434cbd [term 1 FOLLOWER]: Raft consensus is shut down!
W20260620 11:42:35.301578 8805 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260620 11:42:35.301966 9351 tablet_replica.cc:330] T fa7dc55b71324dd2be32fd47a168d402 P 170e23b744f94a62854cb7cd24ac7506: stopping tablet replica
I20260620 11:42:35.301975 9224 tablet_service.cc:1514] Processing DeleteTablet for tablet fa7dc55b71324dd2be32fd47a168d402 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 11:42:35 UTC) from {username='kudu', principal='kudu/127.6.234.254@KRBTEST.COM'} at 127.0.0.1:45108
I20260620 11:42:35.302094 9351 raft_consensus.cc:2229] T fa7dc55b71324dd2be32fd47a168d402 P 170e23b744f94a62854cb7cd24ac7506 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260620 11:42:35.302165 9351 raft_consensus.cc:2258] T fa7dc55b71324dd2be32fd47a168d402 P 170e23b744f94a62854cb7cd24ac7506 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 11:42:35.302266 9352 tablet_replica.cc:330] T fa7dc55b71324dd2be32fd47a168d402 P 36f86b86532e42758e1f5249c186f526: stopping tablet replica
I20260620 11:42:35.302407 9352 raft_consensus.cc:2229] T fa7dc55b71324dd2be32fd47a168d402 P 36f86b86532e42758e1f5249c186f526 [term 1 LEADER]: Raft consensus shutting down.
I20260620 11:42:35.302554 9352 raft_consensus.cc:2258] T fa7dc55b71324dd2be32fd47a168d402 P 36f86b86532e42758e1f5249c186f526 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 11:42:35.302606 9350 ts_tablet_manager.cc:1905] T fa7dc55b71324dd2be32fd47a168d402 P 47814c4c6447409dbab77568f6434cbd: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 11:42:35.303081 9352 ts_tablet_manager.cc:1905] T fa7dc55b71324dd2be32fd47a168d402 P 36f86b86532e42758e1f5249c186f526: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 11:42:35.303474 9084 tablet_service.cc:1467] Processing CreateTablet for tablet 5ffb2448a1a84c4b972bb1a69ff3dd5f (DEFAULT_TABLE table=test-table [id=b1f677cf66684ec4a963c7590213ca36]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 11:42:35.303601 9084 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 5ffb2448a1a84c4b972bb1a69ff3dd5f. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 11:42:35.304894 9350 ts_tablet_manager.cc:1918] T fa7dc55b71324dd2be32fd47a168d402 P 47814c4c6447409dbab77568f6434cbd: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260620 11:42:35.304973 9350 log.cc:1196] T fa7dc55b71324dd2be32fd47a168d402 P 47814c4c6447409dbab77568f6434cbd: Deleting WAL directory at /tmp/dist-test-task1JRg7N/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781955746769051-7083-0/minicluster-data/ts-0/wal/wals/fa7dc55b71324dd2be32fd47a168d402
I20260620 11:42:35.305022 9312 tablet_bootstrap.cc:492] T 5ffb2448a1a84c4b972bb1a69ff3dd5f P 170e23b744f94a62854cb7cd24ac7506: Bootstrap starting.
I20260620 11:42:35.304939 9352 ts_tablet_manager.cc:1918] T fa7dc55b71324dd2be32fd47a168d402 P 36f86b86532e42758e1f5249c186f526: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260620 11:42:35.305210 9352 log.cc:1196] T fa7dc55b71324dd2be32fd47a168d402 P 36f86b86532e42758e1f5249c186f526: Deleting WAL directory at /tmp/dist-test-task1JRg7N/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781955746769051-7083-0/minicluster-data/ts-2/wal/wals/fa7dc55b71324dd2be32fd47a168d402
I20260620 11:42:35.305279 9350 ts_tablet_manager.cc:1939] T fa7dc55b71324dd2be32fd47a168d402 P 47814c4c6447409dbab77568f6434cbd: Deleting consensus metadata
I20260620 11:42:35.305568 9352 ts_tablet_manager.cc:1939] T fa7dc55b71324dd2be32fd47a168d402 P 36f86b86532e42758e1f5249c186f526: Deleting consensus metadata
I20260620 11:42:35.305748 9312 tablet_bootstrap.cc:654] T 5ffb2448a1a84c4b972bb1a69ff3dd5f P 170e23b744f94a62854cb7cd24ac7506: Neither blocks nor log segments found. Creating new log.
I20260620 11:42:35.305891 8944 tablet_service.cc:1467] Processing CreateTablet for tablet 5ffb2448a1a84c4b972bb1a69ff3dd5f (DEFAULT_TABLE table=test-table [id=b1f677cf66684ec4a963c7590213ca36]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 11:42:35.305996 8944 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 5ffb2448a1a84c4b972bb1a69ff3dd5f. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 11:42:35.306185 8791 catalog_manager.cc:4841] TS 47814c4c6447409dbab77568f6434cbd (127.6.234.193:40543): tablet fa7dc55b71324dd2be32fd47a168d402 (table test-table [id=52b490d05bc34359be17ea89ded471b9]) successfully deleted
I20260620 11:42:35.306288 8793 catalog_manager.cc:4841] TS 36f86b86532e42758e1f5249c186f526 (127.6.234.195:37197): tablet fa7dc55b71324dd2be32fd47a168d402 (table test-table [id=52b490d05bc34359be17ea89ded471b9]) successfully deleted
I20260620 11:42:35.306486 9312 tablet_bootstrap.cc:492] T 5ffb2448a1a84c4b972bb1a69ff3dd5f P 170e23b744f94a62854cb7cd24ac7506: No bootstrap required, opened a new log
I20260620 11:42:35.306542 9312 ts_tablet_manager.cc:1397] T 5ffb2448a1a84c4b972bb1a69ff3dd5f P 170e23b744f94a62854cb7cd24ac7506: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260620 11:42:35.306692 9312 raft_consensus.cc:348] T 5ffb2448a1a84c4b972bb1a69ff3dd5f P 170e23b744f94a62854cb7cd24ac7506 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "47814c4c6447409dbab77568f6434cbd" member_type: VOTER last_known_addr { host: "127.6.234.193" port: 40543 } } peers { permanent_uuid: "170e23b744f94a62854cb7cd24ac7506" member_type: VOTER last_known_addr { host: "127.6.234.194" port: 44189 } } peers { permanent_uuid: "36f86b86532e42758e1f5249c186f526" member_type: VOTER last_known_addr { host: "127.6.234.195" port: 37197 } }
I20260620 11:42:35.306777 9312 raft_consensus.cc:374] T 5ffb2448a1a84c4b972bb1a69ff3dd5f P 170e23b744f94a62854cb7cd24ac7506 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 11:42:35.306800 9312 raft_consensus.cc:729] T 5ffb2448a1a84c4b972bb1a69ff3dd5f P 170e23b744f94a62854cb7cd24ac7506 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 170e23b744f94a62854cb7cd24ac7506, State: Initialized, Role: FOLLOWER
I20260620 11:42:35.306844 9312 consensus_queue.cc:260] T 5ffb2448a1a84c4b972bb1a69ff3dd5f P 170e23b744f94a62854cb7cd24ac7506 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "47814c4c6447409dbab77568f6434cbd" member_type: VOTER last_known_addr { host: "127.6.234.193" port: 40543 } } peers { permanent_uuid: "170e23b744f94a62854cb7cd24ac7506" member_type: VOTER last_known_addr { host: "127.6.234.194" port: 44189 } } peers { permanent_uuid: "36f86b86532e42758e1f5249c186f526" member_type: VOTER last_known_addr { host: "127.6.234.195" port: 37197 } }
I20260620 11:42:35.306931 9312 ts_tablet_manager.cc:1428] T 5ffb2448a1a84c4b972bb1a69ff3dd5f P 170e23b744f94a62854cb7cd24ac7506: Time spent starting tablet: real 0.000s user 0.001s sys 0.000s
I20260620 11:42:35.307264 9310 tablet_bootstrap.cc:492] T 5ffb2448a1a84c4b972bb1a69ff3dd5f P 47814c4c6447409dbab77568f6434cbd: Bootstrap starting.
I20260620 11:42:35.307435 9224 tablet_service.cc:1467] Processing CreateTablet for tablet 5ffb2448a1a84c4b972bb1a69ff3dd5f (DEFAULT_TABLE table=test-table [id=b1f677cf66684ec4a963c7590213ca36]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 11:42:35.307540 9224 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 5ffb2448a1a84c4b972bb1a69ff3dd5f. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 11:42:35.307643 9310 tablet_bootstrap.cc:654] T 5ffb2448a1a84c4b972bb1a69ff3dd5f P 47814c4c6447409dbab77568f6434cbd: Neither blocks nor log segments found. Creating new log.
I20260620 11:42:35.308632 9311 tablet_bootstrap.cc:492] T 5ffb2448a1a84c4b972bb1a69ff3dd5f P 36f86b86532e42758e1f5249c186f526: Bootstrap starting.
I20260620 11:42:35.308970 9351 ts_tablet_manager.cc:1905] T fa7dc55b71324dd2be32fd47a168d402 P 170e23b744f94a62854cb7cd24ac7506: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 11:42:35.309015 9311 tablet_bootstrap.cc:654] T 5ffb2448a1a84c4b972bb1a69ff3dd5f P 36f86b86532e42758e1f5249c186f526: Neither blocks nor log segments found. Creating new log.
I20260620 11:42:35.309767 9311 tablet_bootstrap.cc:492] T 5ffb2448a1a84c4b972bb1a69ff3dd5f P 36f86b86532e42758e1f5249c186f526: No bootstrap required, opened a new log
I20260620 11:42:35.309819 9311 ts_tablet_manager.cc:1397] T 5ffb2448a1a84c4b972bb1a69ff3dd5f P 36f86b86532e42758e1f5249c186f526: Time spent bootstrapping tablet: real 0.001s user 0.000s sys 0.001s
I20260620 11:42:35.309958 9311 raft_consensus.cc:348] T 5ffb2448a1a84c4b972bb1a69ff3dd5f P 36f86b86532e42758e1f5249c186f526 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "47814c4c6447409dbab77568f6434cbd" member_type: VOTER last_known_addr { host: "127.6.234.193" port: 40543 } } peers { permanent_uuid: "170e23b744f94a62854cb7cd24ac7506" member_type: VOTER last_known_addr { host: "127.6.234.194" port: 44189 } } peers { permanent_uuid: "36f86b86532e42758e1f5249c186f526" member_type: VOTER last_known_addr { host: "127.6.234.195" port: 37197 } }
I20260620 11:42:35.310029 9311 raft_consensus.cc:374] T 5ffb2448a1a84c4b972bb1a69ff3dd5f P 36f86b86532e42758e1f5249c186f526 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 11:42:35.310048 9311 raft_consensus.cc:729] T 5ffb2448a1a84c4b972bb1a69ff3dd5f P 36f86b86532e42758e1f5249c186f526 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 36f86b86532e42758e1f5249c186f526, State: Initialized, Role: FOLLOWER
I20260620 11:42:35.310087 9311 consensus_queue.cc:260] T 5ffb2448a1a84c4b972bb1a69ff3dd5f P 36f86b86532e42758e1f5249c186f526 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "47814c4c6447409dbab77568f6434cbd" member_type: VOTER last_known_addr { host: "127.6.234.193" port: 40543 } } peers { permanent_uuid: "170e23b744f94a62854cb7cd24ac7506" member_type: VOTER last_known_addr { host: "127.6.234.194" port: 44189 } } peers { permanent_uuid: "36f86b86532e42758e1f5249c186f526" member_type: VOTER last_known_addr { host: "127.6.234.195" port: 37197 } }
I20260620 11:42:35.310170 9311 ts_tablet_manager.cc:1428] T 5ffb2448a1a84c4b972bb1a69ff3dd5f P 36f86b86532e42758e1f5249c186f526: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
I20260620 11:42:35.310529 9351 ts_tablet_manager.cc:1918] T fa7dc55b71324dd2be32fd47a168d402 P 170e23b744f94a62854cb7cd24ac7506: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260620 11:42:35.310600 9351 log.cc:1196] T fa7dc55b71324dd2be32fd47a168d402 P 170e23b744f94a62854cb7cd24ac7506: Deleting WAL directory at /tmp/dist-test-task1JRg7N/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781955746769051-7083-0/minicluster-data/ts-1/wal/wals/fa7dc55b71324dd2be32fd47a168d402
I20260620 11:42:35.310855 9351 ts_tablet_manager.cc:1939] T fa7dc55b71324dd2be32fd47a168d402 P 170e23b744f94a62854cb7cd24ac7506: Deleting consensus metadata
I20260620 11:42:35.311244 8793 catalog_manager.cc:4841] TS 170e23b744f94a62854cb7cd24ac7506 (127.6.234.194:44189): tablet fa7dc55b71324dd2be32fd47a168d402 (table test-table [id=52b490d05bc34359be17ea89ded471b9]) successfully deleted
I20260620 11:42:35.314093 9310 tablet_bootstrap.cc:492] T 5ffb2448a1a84c4b972bb1a69ff3dd5f P 47814c4c6447409dbab77568f6434cbd: No bootstrap required, opened a new log
I20260620 11:42:35.314160 9310 ts_tablet_manager.cc:1397] T 5ffb2448a1a84c4b972bb1a69ff3dd5f P 47814c4c6447409dbab77568f6434cbd: Time spent bootstrapping tablet: real 0.007s user 0.001s sys 0.000s
I20260620 11:42:35.314330 9310 raft_consensus.cc:348] T 5ffb2448a1a84c4b972bb1a69ff3dd5f P 47814c4c6447409dbab77568f6434cbd [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "47814c4c6447409dbab77568f6434cbd" member_type: VOTER last_known_addr { host: "127.6.234.193" port: 40543 } } peers { permanent_uuid: "170e23b744f94a62854cb7cd24ac7506" member_type: VOTER last_known_addr { host: "127.6.234.194" port: 44189 } } peers { permanent_uuid: "36f86b86532e42758e1f5249c186f526" member_type: VOTER last_known_addr { host: "127.6.234.195" port: 37197 } }
I20260620 11:42:35.314405 9310 raft_consensus.cc:374] T 5ffb2448a1a84c4b972bb1a69ff3dd5f P 47814c4c6447409dbab77568f6434cbd [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 11:42:35.314437 9310 raft_consensus.cc:729] T 5ffb2448a1a84c4b972bb1a69ff3dd5f P 47814c4c6447409dbab77568f6434cbd [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 47814c4c6447409dbab77568f6434cbd, State: Initialized, Role: FOLLOWER
I20260620 11:42:35.314489 9310 consensus_queue.cc:260] T 5ffb2448a1a84c4b972bb1a69ff3dd5f P 47814c4c6447409dbab77568f6434cbd [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "47814c4c6447409dbab77568f6434cbd" member_type: VOTER last_known_addr { host: "127.6.234.193" port: 40543 } } peers { permanent_uuid: "170e23b744f94a62854cb7cd24ac7506" member_type: VOTER last_known_addr { host: "127.6.234.194" port: 44189 } } peers { permanent_uuid: "36f86b86532e42758e1f5249c186f526" member_type: VOTER last_known_addr { host: "127.6.234.195" port: 37197 } }
I20260620 11:42:35.314602 9310 ts_tablet_manager.cc:1428] T 5ffb2448a1a84c4b972bb1a69ff3dd5f P 47814c4c6447409dbab77568f6434cbd: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
I20260620 11:42:35.336527 9320 raft_consensus.cc:482] T 5ffb2448a1a84c4b972bb1a69ff3dd5f P 36f86b86532e42758e1f5249c186f526 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260620 11:42:35.336644 9320 raft_consensus.cc:504] T 5ffb2448a1a84c4b972bb1a69ff3dd5f P 36f86b86532e42758e1f5249c186f526 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "47814c4c6447409dbab77568f6434cbd" member_type: VOTER last_known_addr { host: "127.6.234.193" port: 40543 } } peers { permanent_uuid: "170e23b744f94a62854cb7cd24ac7506" member_type: VOTER last_known_addr { host: "127.6.234.194" port: 44189 } } peers { permanent_uuid: "36f86b86532e42758e1f5249c186f526" member_type: VOTER last_known_addr { host: "127.6.234.195" port: 37197 } }
I20260620 11:42:35.336851 9320 leader_election.cc:288] T 5ffb2448a1a84c4b972bb1a69ff3dd5f P 36f86b86532e42758e1f5249c186f526 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 47814c4c6447409dbab77568f6434cbd (127.6.234.193:40543), 170e23b744f94a62854cb7cd24ac7506 (127.6.234.194:44189)
I20260620 11:42:35.337071 8964 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5ffb2448a1a84c4b972bb1a69ff3dd5f" candidate_uuid: "36f86b86532e42758e1f5249c186f526" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "47814c4c6447409dbab77568f6434cbd" is_pre_election: true
I20260620 11:42:35.337073 9104 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5ffb2448a1a84c4b972bb1a69ff3dd5f" candidate_uuid: "36f86b86532e42758e1f5249c186f526" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "170e23b744f94a62854cb7cd24ac7506" is_pre_election: true
I20260620 11:42:35.337172 8964 raft_consensus.cc:2454] T 5ffb2448a1a84c4b972bb1a69ff3dd5f P 47814c4c6447409dbab77568f6434cbd [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 36f86b86532e42758e1f5249c186f526 in term 0.
I20260620 11:42:35.337172 9104 raft_consensus.cc:2454] T 5ffb2448a1a84c4b972bb1a69ff3dd5f P 170e23b744f94a62854cb7cd24ac7506 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 36f86b86532e42758e1f5249c186f526 in term 0.
I20260620 11:42:35.337329 9177 leader_election.cc:302] T 5ffb2448a1a84c4b972bb1a69ff3dd5f P 36f86b86532e42758e1f5249c186f526 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 170e23b744f94a62854cb7cd24ac7506, 36f86b86532e42758e1f5249c186f526; no voters:
I20260620 11:42:35.337421 9320 raft_consensus.cc:2781] T 5ffb2448a1a84c4b972bb1a69ff3dd5f P 36f86b86532e42758e1f5249c186f526 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260620 11:42:35.337455 9320 raft_consensus.cc:482] T 5ffb2448a1a84c4b972bb1a69ff3dd5f P 36f86b86532e42758e1f5249c186f526 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260620 11:42:35.337469 9320 raft_consensus.cc:3037] T 5ffb2448a1a84c4b972bb1a69ff3dd5f P 36f86b86532e42758e1f5249c186f526 [term 0 FOLLOWER]: Advancing to term 1
I20260620 11:42:35.338208 9320 raft_consensus.cc:504] T 5ffb2448a1a84c4b972bb1a69ff3dd5f P 36f86b86532e42758e1f5249c186f526 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "47814c4c6447409dbab77568f6434cbd" member_type: VOTER last_known_addr { host: "127.6.234.193" port: 40543 } } peers { permanent_uuid: "170e23b744f94a62854cb7cd24ac7506" member_type: VOTER last_known_addr { host: "127.6.234.194" port: 44189 } } peers { permanent_uuid: "36f86b86532e42758e1f5249c186f526" member_type: VOTER last_known_addr { host: "127.6.234.195" port: 37197 } }
I20260620 11:42:35.338338 9320 leader_election.cc:288] T 5ffb2448a1a84c4b972bb1a69ff3dd5f P 36f86b86532e42758e1f5249c186f526 [CANDIDATE]: Term 1 election: Requested vote from peers 47814c4c6447409dbab77568f6434cbd (127.6.234.193:40543), 170e23b744f94a62854cb7cd24ac7506 (127.6.234.194:44189)
I20260620 11:42:35.338474 8964 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5ffb2448a1a84c4b972bb1a69ff3dd5f" candidate_uuid: "36f86b86532e42758e1f5249c186f526" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "47814c4c6447409dbab77568f6434cbd"
I20260620 11:42:35.338546 8964 raft_consensus.cc:3037] T 5ffb2448a1a84c4b972bb1a69ff3dd5f P 47814c4c6447409dbab77568f6434cbd [term 0 FOLLOWER]: Advancing to term 1
I20260620 11:42:35.338541 9104 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5ffb2448a1a84c4b972bb1a69ff3dd5f" candidate_uuid: "36f86b86532e42758e1f5249c186f526" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "170e23b744f94a62854cb7cd24ac7506"
I20260620 11:42:35.338617 9104 raft_consensus.cc:3037] T 5ffb2448a1a84c4b972bb1a69ff3dd5f P 170e23b744f94a62854cb7cd24ac7506 [term 0 FOLLOWER]: Advancing to term 1
I20260620 11:42:35.339354 8964 raft_consensus.cc:2454] T 5ffb2448a1a84c4b972bb1a69ff3dd5f P 47814c4c6447409dbab77568f6434cbd [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 36f86b86532e42758e1f5249c186f526 in term 1.
I20260620 11:42:35.339354 9104 raft_consensus.cc:2454] T 5ffb2448a1a84c4b972bb1a69ff3dd5f P 170e23b744f94a62854cb7cd24ac7506 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 36f86b86532e42758e1f5249c186f526 in term 1.
I20260620 11:42:35.339519 9177 leader_election.cc:302] T 5ffb2448a1a84c4b972bb1a69ff3dd5f P 36f86b86532e42758e1f5249c186f526 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 170e23b744f94a62854cb7cd24ac7506, 36f86b86532e42758e1f5249c186f526; no voters:
I20260620 11:42:35.339632 9320 raft_consensus.cc:2781] T 5ffb2448a1a84c4b972bb1a69ff3dd5f P 36f86b86532e42758e1f5249c186f526 [term 1 FOLLOWER]: Leader election won for term 1
I20260620 11:42:35.339681 9320 raft_consensus.cc:686] T 5ffb2448a1a84c4b972bb1a69ff3dd5f P 36f86b86532e42758e1f5249c186f526 [term 1 LEADER]: Becoming Leader. State: Replica: 36f86b86532e42758e1f5249c186f526, State: Running, Role: LEADER
I20260620 11:42:35.339732 9320 consensus_queue.cc:237] T 5ffb2448a1a84c4b972bb1a69ff3dd5f P 36f86b86532e42758e1f5249c186f526 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "47814c4c6447409dbab77568f6434cbd" member_type: VOTER last_known_addr { host: "127.6.234.193" port: 40543 } } peers { permanent_uuid: "170e23b744f94a62854cb7cd24ac7506" member_type: VOTER last_known_addr { host: "127.6.234.194" port: 44189 } } peers { permanent_uuid: "36f86b86532e42758e1f5249c186f526" member_type: VOTER last_known_addr { host: "127.6.234.195" port: 37197 } }
I20260620 11:42:35.340317 8805 catalog_manager.cc:5495] T 5ffb2448a1a84c4b972bb1a69ff3dd5f P 36f86b86532e42758e1f5249c186f526 reported cstate change: term changed from 0 to 1, leader changed from <none> to 36f86b86532e42758e1f5249c186f526 (127.6.234.195). New cstate: current_term: 1 leader_uuid: "36f86b86532e42758e1f5249c186f526" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "47814c4c6447409dbab77568f6434cbd" member_type: VOTER last_known_addr { host: "127.6.234.193" port: 40543 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "170e23b744f94a62854cb7cd24ac7506" member_type: VOTER last_known_addr { host: "127.6.234.194" port: 44189 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "36f86b86532e42758e1f5249c186f526" member_type: VOTER last_known_addr { host: "127.6.234.195" port: 37197 } health_report { overall_health: HEALTHY } } }
I20260620 11:42:35.347513 8964 raft_consensus.cc:1261] T 5ffb2448a1a84c4b972bb1a69ff3dd5f P 47814c4c6447409dbab77568f6434cbd [term 1 FOLLOWER]: Refusing update from remote peer 36f86b86532e42758e1f5249c186f526: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 11:42:35.347528 9104 raft_consensus.cc:1261] T 5ffb2448a1a84c4b972bb1a69ff3dd5f P 170e23b744f94a62854cb7cd24ac7506 [term 1 FOLLOWER]: Refusing update from remote peer 36f86b86532e42758e1f5249c186f526: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 11:42:35.347718 9346 consensus_queue.cc:1035] T 5ffb2448a1a84c4b972bb1a69ff3dd5f P 36f86b86532e42758e1f5249c186f526 [LEADER]: Connected to new peer: Peer: permanent_uuid: "170e23b744f94a62854cb7cd24ac7506" member_type: VOTER last_known_addr { host: "127.6.234.194" port: 44189 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 11:42:35.347846 9318 consensus_queue.cc:1035] T 5ffb2448a1a84c4b972bb1a69ff3dd5f P 36f86b86532e42758e1f5249c186f526 [LEADER]: Connected to new peer: Peer: permanent_uuid: "47814c4c6447409dbab77568f6434cbd" member_type: VOTER last_known_addr { host: "127.6.234.193" port: 40543 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 11:42:35.348845 9335 mvcc.cc:205] Tried to move back new op lower bound from 7298890773902233600 to 7298890773871947776. Current Snapshot: MvccSnapshot[applied={T|T < 7298890773902233600}]
W20260620 11:42:35.352605 9335 tablet_replica.cc:1298] Aborted: operation has been aborted: cancelling pending write operations
I20260620 11:42:36.870191 8806 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:47676:
table { table_name: "test-table" } modify_external_catalogs: true
I20260620 11:42:36.870306 8806 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:47676:
table { table_name: "test-table" } modify_external_catalogs: true
I20260620 11:42:36.871153 8806 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 6e89f37d51dc404c95261d264710ef80: Sending DeleteTablet for 3 replicas of tablet 5ffb2448a1a84c4b972bb1a69ff3dd5f
I20260620 11:42:36.871455 9223 tablet_service.cc:1514] Processing DeleteTablet for tablet 5ffb2448a1a84c4b972bb1a69ff3dd5f with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 11:42:36 UTC) from {username='kudu', principal='kudu/127.6.234.254@KRBTEST.COM'} at 127.0.0.1:45108
I20260620 11:42:36.871462 8944 tablet_service.cc:1514] Processing DeleteTablet for tablet 5ffb2448a1a84c4b972bb1a69ff3dd5f with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 11:42:36 UTC) from {username='kudu', principal='kudu/127.6.234.254@KRBTEST.COM'} at 127.0.0.1:58340
I20260620 11:42:36.871467 9084 tablet_service.cc:1514] Processing DeleteTablet for tablet 5ffb2448a1a84c4b972bb1a69ff3dd5f with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 11:42:36 UTC) from {username='kudu', principal='kudu/127.6.234.254@KRBTEST.COM'} at 127.0.0.1:53692
I20260620 11:42:36.871647 9379 tablet_replica.cc:330] T 5ffb2448a1a84c4b972bb1a69ff3dd5f P 36f86b86532e42758e1f5249c186f526: stopping tablet replica
I20260620 11:42:36.871685 9380 tablet_replica.cc:330] T 5ffb2448a1a84c4b972bb1a69ff3dd5f P 170e23b744f94a62854cb7cd24ac7506: stopping tablet replica
I20260620 11:42:36.871709 9379 raft_consensus.cc:2229] T 5ffb2448a1a84c4b972bb1a69ff3dd5f P 36f86b86532e42758e1f5249c186f526 [term 1 LEADER]: Raft consensus shutting down.
I20260620 11:42:36.871737 9380 raft_consensus.cc:2229] T 5ffb2448a1a84c4b972bb1a69ff3dd5f P 170e23b744f94a62854cb7cd24ac7506 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260620 11:42:36.871774 9380 raft_consensus.cc:2258] T 5ffb2448a1a84c4b972bb1a69ff3dd5f P 170e23b744f94a62854cb7cd24ac7506 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 11:42:36.871821 9378 tablet_replica.cc:330] T 5ffb2448a1a84c4b972bb1a69ff3dd5f P 47814c4c6447409dbab77568f6434cbd: stopping tablet replica
I20260620 11:42:36.871879 9378 raft_consensus.cc:2229] T 5ffb2448a1a84c4b972bb1a69ff3dd5f P 47814c4c6447409dbab77568f6434cbd [term 1 FOLLOWER]: Raft consensus shutting down.
I20260620 11:42:36.871927 9378 raft_consensus.cc:2258] T 5ffb2448a1a84c4b972bb1a69ff3dd5f P 47814c4c6447409dbab77568f6434cbd [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 11:42:36.872020 9379 raft_consensus.cc:2258] T 5ffb2448a1a84c4b972bb1a69ff3dd5f P 36f86b86532e42758e1f5249c186f526 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 11:42:36.872044 9380 ts_tablet_manager.cc:1905] T 5ffb2448a1a84c4b972bb1a69ff3dd5f P 170e23b744f94a62854cb7cd24ac7506: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 11:42:36.872262 9378 ts_tablet_manager.cc:1905] T 5ffb2448a1a84c4b972bb1a69ff3dd5f P 47814c4c6447409dbab77568f6434cbd: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 11:42:36.873781 9380 ts_tablet_manager.cc:1918] T 5ffb2448a1a84c4b972bb1a69ff3dd5f P 170e23b744f94a62854cb7cd24ac7506: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.8
I20260620 11:42:36.873850 9380 log.cc:1196] T 5ffb2448a1a84c4b972bb1a69ff3dd5f P 170e23b744f94a62854cb7cd24ac7506: Deleting WAL directory at /tmp/dist-test-task1JRg7N/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781955746769051-7083-0/minicluster-data/ts-1/wal/wals/5ffb2448a1a84c4b972bb1a69ff3dd5f
I20260620 11:42:36.873842 9378 ts_tablet_manager.cc:1918] T 5ffb2448a1a84c4b972bb1a69ff3dd5f P 47814c4c6447409dbab77568f6434cbd: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.8
I20260620 11:42:36.873889 9378 log.cc:1196] T 5ffb2448a1a84c4b972bb1a69ff3dd5f P 47814c4c6447409dbab77568f6434cbd: Deleting WAL directory at /tmp/dist-test-task1JRg7N/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781955746769051-7083-0/minicluster-data/ts-0/wal/wals/5ffb2448a1a84c4b972bb1a69ff3dd5f
I20260620 11:42:36.874086 9380 ts_tablet_manager.cc:1939] T 5ffb2448a1a84c4b972bb1a69ff3dd5f P 170e23b744f94a62854cb7cd24ac7506: Deleting consensus metadata
I20260620 11:42:36.874099 9378 ts_tablet_manager.cc:1939] T 5ffb2448a1a84c4b972bb1a69ff3dd5f P 47814c4c6447409dbab77568f6434cbd: Deleting consensus metadata
I20260620 11:42:36.874467 8791 catalog_manager.cc:4841] TS 47814c4c6447409dbab77568f6434cbd (127.6.234.193:40543): tablet 5ffb2448a1a84c4b972bb1a69ff3dd5f (table test-table [id=b1f677cf66684ec4a963c7590213ca36]) successfully deleted
I20260620 11:42:36.874467 8793 catalog_manager.cc:4841] TS 170e23b744f94a62854cb7cd24ac7506 (127.6.234.194:44189): tablet 5ffb2448a1a84c4b972bb1a69ff3dd5f (table test-table [id=b1f677cf66684ec4a963c7590213ca36]) successfully deleted
I20260620 11:42:36.874611 9379 ts_tablet_manager.cc:1905] T 5ffb2448a1a84c4b972bb1a69ff3dd5f P 36f86b86532e42758e1f5249c186f526: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 11:42:36.875933 9379 ts_tablet_manager.cc:1918] T 5ffb2448a1a84c4b972bb1a69ff3dd5f P 36f86b86532e42758e1f5249c186f526: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.8
I20260620 11:42:36.875998 9379 log.cc:1196] T 5ffb2448a1a84c4b972bb1a69ff3dd5f P 36f86b86532e42758e1f5249c186f526: Deleting WAL directory at /tmp/dist-test-task1JRg7N/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781955746769051-7083-0/minicluster-data/ts-2/wal/wals/5ffb2448a1a84c4b972bb1a69ff3dd5f
I20260620 11:42:36.876257 9379 ts_tablet_manager.cc:1939] T 5ffb2448a1a84c4b972bb1a69ff3dd5f P 36f86b86532e42758e1f5249c186f526: Deleting consensus metadata
I20260620 11:42:36.876597 8793 catalog_manager.cc:4841] TS 36f86b86532e42758e1f5249c186f526 (127.6.234.195:37197): tablet 5ffb2448a1a84c4b972bb1a69ff3dd5f (table test-table [id=b1f677cf66684ec4a963c7590213ca36]) successfully deleted
Jun 20 11:42:36 dist-test-slave-cjc6 krb5kdc[8762](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781955756, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-user@KRBTEST.COM:
Jun 20 11:42:36 dist-test-slave-cjc6 krb5kdc[8762](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781955756, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for kudu/127.6.234.254@KRBTEST.COM
I20260620 11:42:36.901312 9388 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 11:42:36.889345 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47686 (local address 127.6.234.254:36185)
0620 11:42:36.889487 (+ 142us) server_negotiation.cc:206] Beginning negotiation
0620 11:42:36.889489 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 11:42:36.889498 (+ 9us) server_negotiation.cc:407] Connection header received
0620 11:42:36.889542 (+ 44us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 11:42:36.889543 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 11:42:36.889554 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 11:42:36.889614 (+ 60us) server_negotiation.cc:226] Negotiated authn=SASL
0620 11:42:36.890200 (+ 586us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 11:42:36.890701 (+ 501us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 11:42:36.891468 (+ 767us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 11:42:36.891659 (+ 191us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 11:42:36.891710 (+ 51us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 11:42:36.898413 (+ 6703us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 11:42:36.898415 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 11:42:36.898415 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 11:42:36.898422 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0620 11:42:36.900172 (+ 1750us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 11:42:36.900483 (+ 311us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 11:42:36.900484 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 11:42:36.900484 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 11:42:36.900534 (+ 50us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 11:42:36.900845 (+ 311us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 11:42:36.900845 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 11:42:36.900846 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 11:42:36.900948 (+ 102us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 11:42:36.901057 (+ 109us) server_negotiation.cc:1035] Waiting for connection context
0620 11:42:36.901142 (+ 85us) server_negotiation.cc:299] Negotiation successful
0620 11:42:36.901204 (+ 62us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":111,"thread_start_us":80,"threads_started":1}
I20260620 11:42:36.902748 8806 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:47686:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260620 11:42:36.902932 8806 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260620 11:42:36.904264 9084 tablet_service.cc:1467] Processing CreateTablet for tablet 6100d64b207048afabe58fc85b390757 (DEFAULT_TABLE table=test-table [id=35a9ca682f3e45f397df98ddb835f55c]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 11:42:36.904273 9223 tablet_service.cc:1467] Processing CreateTablet for tablet 6100d64b207048afabe58fc85b390757 (DEFAULT_TABLE table=test-table [id=35a9ca682f3e45f397df98ddb835f55c]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 11:42:36.904269 8944 tablet_service.cc:1467] Processing CreateTablet for tablet 6100d64b207048afabe58fc85b390757 (DEFAULT_TABLE table=test-table [id=35a9ca682f3e45f397df98ddb835f55c]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 11:42:36.904420 9084 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 6100d64b207048afabe58fc85b390757. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 11:42:36.904420 9223 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 6100d64b207048afabe58fc85b390757. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 11:42:36.904424 8944 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 6100d64b207048afabe58fc85b390757. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 11:42:36.905815 9390 tablet_bootstrap.cc:492] T 6100d64b207048afabe58fc85b390757 P 47814c4c6447409dbab77568f6434cbd: Bootstrap starting.
I20260620 11:42:36.905946 9391 tablet_bootstrap.cc:492] T 6100d64b207048afabe58fc85b390757 P 36f86b86532e42758e1f5249c186f526: Bootstrap starting.
I20260620 11:42:36.906199 9390 tablet_bootstrap.cc:654] T 6100d64b207048afabe58fc85b390757 P 47814c4c6447409dbab77568f6434cbd: Neither blocks nor log segments found. Creating new log.
I20260620 11:42:36.906311 9391 tablet_bootstrap.cc:654] T 6100d64b207048afabe58fc85b390757 P 36f86b86532e42758e1f5249c186f526: Neither blocks nor log segments found. Creating new log.
I20260620 11:42:36.906318 9389 tablet_bootstrap.cc:492] T 6100d64b207048afabe58fc85b390757 P 170e23b744f94a62854cb7cd24ac7506: Bootstrap starting.
I20260620 11:42:36.906659 9389 tablet_bootstrap.cc:654] T 6100d64b207048afabe58fc85b390757 P 170e23b744f94a62854cb7cd24ac7506: Neither blocks nor log segments found. Creating new log.
I20260620 11:42:36.906800 9390 tablet_bootstrap.cc:492] T 6100d64b207048afabe58fc85b390757 P 47814c4c6447409dbab77568f6434cbd: No bootstrap required, opened a new log
I20260620 11:42:36.906857 9390 ts_tablet_manager.cc:1397] T 6100d64b207048afabe58fc85b390757 P 47814c4c6447409dbab77568f6434cbd: Time spent bootstrapping tablet: real 0.001s user 0.001s sys 0.000s
I20260620 11:42:36.907035 9390 raft_consensus.cc:348] T 6100d64b207048afabe58fc85b390757 P 47814c4c6447409dbab77568f6434cbd [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "47814c4c6447409dbab77568f6434cbd" member_type: VOTER last_known_addr { host: "127.6.234.193" port: 40543 } } peers { permanent_uuid: "170e23b744f94a62854cb7cd24ac7506" member_type: VOTER last_known_addr { host: "127.6.234.194" port: 44189 } } peers { permanent_uuid: "36f86b86532e42758e1f5249c186f526" member_type: VOTER last_known_addr { host: "127.6.234.195" port: 37197 } }
I20260620 11:42:36.907146 9390 raft_consensus.cc:374] T 6100d64b207048afabe58fc85b390757 P 47814c4c6447409dbab77568f6434cbd [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 11:42:36.907168 9390 raft_consensus.cc:729] T 6100d64b207048afabe58fc85b390757 P 47814c4c6447409dbab77568f6434cbd [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 47814c4c6447409dbab77568f6434cbd, State: Initialized, Role: FOLLOWER
I20260620 11:42:36.907322 9389 tablet_bootstrap.cc:492] T 6100d64b207048afabe58fc85b390757 P 170e23b744f94a62854cb7cd24ac7506: No bootstrap required, opened a new log
I20260620 11:42:36.907369 9391 tablet_bootstrap.cc:492] T 6100d64b207048afabe58fc85b390757 P 36f86b86532e42758e1f5249c186f526: No bootstrap required, opened a new log
I20260620 11:42:36.907388 9389 ts_tablet_manager.cc:1397] T 6100d64b207048afabe58fc85b390757 P 170e23b744f94a62854cb7cd24ac7506: Time spent bootstrapping tablet: real 0.001s user 0.001s sys 0.000s
I20260620 11:42:36.907428 9391 ts_tablet_manager.cc:1397] T 6100d64b207048afabe58fc85b390757 P 36f86b86532e42758e1f5249c186f526: Time spent bootstrapping tablet: real 0.001s user 0.001s sys 0.000s
I20260620 11:42:36.907537 9389 raft_consensus.cc:348] T 6100d64b207048afabe58fc85b390757 P 170e23b744f94a62854cb7cd24ac7506 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "47814c4c6447409dbab77568f6434cbd" member_type: VOTER last_known_addr { host: "127.6.234.193" port: 40543 } } peers { permanent_uuid: "170e23b744f94a62854cb7cd24ac7506" member_type: VOTER last_known_addr { host: "127.6.234.194" port: 44189 } } peers { permanent_uuid: "36f86b86532e42758e1f5249c186f526" member_type: VOTER last_known_addr { host: "127.6.234.195" port: 37197 } }
I20260620 11:42:36.907543 9391 raft_consensus.cc:348] T 6100d64b207048afabe58fc85b390757 P 36f86b86532e42758e1f5249c186f526 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "47814c4c6447409dbab77568f6434cbd" member_type: VOTER last_known_addr { host: "127.6.234.193" port: 40543 } } peers { permanent_uuid: "170e23b744f94a62854cb7cd24ac7506" member_type: VOTER last_known_addr { host: "127.6.234.194" port: 44189 } } peers { permanent_uuid: "36f86b86532e42758e1f5249c186f526" member_type: VOTER last_known_addr { host: "127.6.234.195" port: 37197 } }
I20260620 11:42:36.907603 9391 raft_consensus.cc:374] T 6100d64b207048afabe58fc85b390757 P 36f86b86532e42758e1f5249c186f526 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 11:42:36.907617 9391 raft_consensus.cc:729] T 6100d64b207048afabe58fc85b390757 P 36f86b86532e42758e1f5249c186f526 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 36f86b86532e42758e1f5249c186f526, State: Initialized, Role: FOLLOWER
I20260620 11:42:36.907622 9389 raft_consensus.cc:374] T 6100d64b207048afabe58fc85b390757 P 170e23b744f94a62854cb7cd24ac7506 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 11:42:36.907647 9389 raft_consensus.cc:729] T 6100d64b207048afabe58fc85b390757 P 170e23b744f94a62854cb7cd24ac7506 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 170e23b744f94a62854cb7cd24ac7506, State: Initialized, Role: FOLLOWER
I20260620 11:42:36.907670 9391 consensus_queue.cc:260] T 6100d64b207048afabe58fc85b390757 P 36f86b86532e42758e1f5249c186f526 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "47814c4c6447409dbab77568f6434cbd" member_type: VOTER last_known_addr { host: "127.6.234.193" port: 40543 } } peers { permanent_uuid: "170e23b744f94a62854cb7cd24ac7506" member_type: VOTER last_known_addr { host: "127.6.234.194" port: 44189 } } peers { permanent_uuid: "36f86b86532e42758e1f5249c186f526" member_type: VOTER last_known_addr { host: "127.6.234.195" port: 37197 } }
I20260620 11:42:36.907712 9389 consensus_queue.cc:260] T 6100d64b207048afabe58fc85b390757 P 170e23b744f94a62854cb7cd24ac7506 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "47814c4c6447409dbab77568f6434cbd" member_type: VOTER last_known_addr { host: "127.6.234.193" port: 40543 } } peers { permanent_uuid: "170e23b744f94a62854cb7cd24ac7506" member_type: VOTER last_known_addr { host: "127.6.234.194" port: 44189 } } peers { permanent_uuid: "36f86b86532e42758e1f5249c186f526" member_type: VOTER last_known_addr { host: "127.6.234.195" port: 37197 } }
I20260620 11:42:36.907770 9391 ts_tablet_manager.cc:1428] T 6100d64b207048afabe58fc85b390757 P 36f86b86532e42758e1f5249c186f526: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
I20260620 11:42:36.907892 9389 ts_tablet_manager.cc:1428] T 6100d64b207048afabe58fc85b390757 P 170e23b744f94a62854cb7cd24ac7506: Time spent starting tablet: real 0.000s user 0.001s sys 0.000s
I20260620 11:42:36.907882 9390 consensus_queue.cc:260] T 6100d64b207048afabe58fc85b390757 P 47814c4c6447409dbab77568f6434cbd [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "47814c4c6447409dbab77568f6434cbd" member_type: VOTER last_known_addr { host: "127.6.234.193" port: 40543 } } peers { permanent_uuid: "170e23b744f94a62854cb7cd24ac7506" member_type: VOTER last_known_addr { host: "127.6.234.194" port: 44189 } } peers { permanent_uuid: "36f86b86532e42758e1f5249c186f526" member_type: VOTER last_known_addr { host: "127.6.234.195" port: 37197 } }
I20260620 11:42:36.908087 9390 ts_tablet_manager.cc:1428] T 6100d64b207048afabe58fc85b390757 P 47814c4c6447409dbab77568f6434cbd: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20260620 11:42:36.999141 9395 raft_consensus.cc:482] T 6100d64b207048afabe58fc85b390757 P 170e23b744f94a62854cb7cd24ac7506 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260620 11:42:36.999239 9395 raft_consensus.cc:504] T 6100d64b207048afabe58fc85b390757 P 170e23b744f94a62854cb7cd24ac7506 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "47814c4c6447409dbab77568f6434cbd" member_type: VOTER last_known_addr { host: "127.6.234.193" port: 40543 } } peers { permanent_uuid: "170e23b744f94a62854cb7cd24ac7506" member_type: VOTER last_known_addr { host: "127.6.234.194" port: 44189 } } peers { permanent_uuid: "36f86b86532e42758e1f5249c186f526" member_type: VOTER last_known_addr { host: "127.6.234.195" port: 37197 } }
I20260620 11:42:36.999656 9395 leader_election.cc:288] T 6100d64b207048afabe58fc85b390757 P 170e23b744f94a62854cb7cd24ac7506 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 47814c4c6447409dbab77568f6434cbd (127.6.234.193:40543), 36f86b86532e42758e1f5249c186f526 (127.6.234.195:37197)
I20260620 11:42:37.002971 9399 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 11:42:36.999883 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.234.193:40543 (local address 127.6.234.194:36471)
0620 11:42:37.000204 (+ 321us) negotiation.cc:107] Waiting for socket to connect
0620 11:42:37.000216 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0620 11:42:37.000272 (+ 56us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 11:42:37.000434 (+ 162us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 11:42:37.000435 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 11:42:37.000437 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 11:42:37.000527 (+ 90us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 11:42:37.000529 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 11:42:37.001217 (+ 688us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 11:42:37.001218 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 11:42:37.002096 (+ 878us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 11:42:37.002098 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 11:42:37.002694 (+ 596us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 11:42:37.002695 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 11:42:37.002764 (+ 69us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 11:42:37.002767 (+ 3us) client_negotiation.cc:769] Sending connection context
0620 11:42:37.002832 (+ 65us) client_negotiation.cc:240] Negotiation successful
0620 11:42:37.002882 (+ 50us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":301,"mutex_wait_us":6,"thread_start_us":141,"threads_started":1}
I20260620 11:42:37.002971 9398 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 11:42:36.999931 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.6.234.194:36471 (local address 127.6.234.193:40543)
0620 11:42:37.000251 (+ 320us) server_negotiation.cc:206] Beginning negotiation
0620 11:42:37.000253 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 11:42:37.000261 (+ 8us) server_negotiation.cc:407] Connection header received
0620 11:42:37.000334 (+ 73us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 11:42:37.000335 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 11:42:37.000345 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 11:42:37.000434 (+ 89us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 11:42:37.000605 (+ 171us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 11:42:37.001146 (+ 541us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 11:42:37.002188 (+ 1042us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 11:42:37.002597 (+ 409us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 11:42:37.002686 (+ 89us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 11:42:37.002695 (+ 9us) server_negotiation.cc:1035] Waiting for connection context
0620 11:42:37.002839 (+ 144us) server_negotiation.cc:299] Negotiation successful
0620 11:42:37.002879 (+ 40us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":291,"thread_start_us":96,"threads_started":1}
I20260620 11:42:37.003151 9397 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 11:42:36.999880 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.234.195:37197 (local address 127.6.234.194:51889)
0620 11:42:37.000174 (+ 294us) negotiation.cc:107] Waiting for socket to connect
0620 11:42:37.000186 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0620 11:42:37.000252 (+ 66us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 11:42:37.000430 (+ 178us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 11:42:37.000431 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 11:42:37.000433 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 11:42:37.000771 (+ 338us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 11:42:37.000773 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 11:42:37.001478 (+ 705us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 11:42:37.001479 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 11:42:37.002283 (+ 804us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 11:42:37.002285 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 11:42:37.002834 (+ 549us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 11:42:37.002835 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 11:42:37.002912 (+ 77us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 11:42:37.002916 (+ 4us) client_negotiation.cc:769] Sending connection context
0620 11:42:37.002993 (+ 77us) client_negotiation.cc:240] Negotiation successful
0620 11:42:37.003029 (+ 36us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":255,"mutex_wait_us":7,"thread_start_us":151,"threads_started":1}
I20260620 11:42:37.003255 9400 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 11:42:36.999942 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.6.234.194:51889 (local address 127.6.234.195:37197)
0620 11:42:37.000175 (+ 233us) server_negotiation.cc:206] Beginning negotiation
0620 11:42:37.000178 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0620 11:42:37.000236 (+ 58us) server_negotiation.cc:407] Connection header received
0620 11:42:37.000318 (+ 82us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 11:42:37.000319 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 11:42:37.000329 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 11:42:37.000425 (+ 96us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 11:42:37.000854 (+ 429us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 11:42:37.001386 (+ 532us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 11:42:37.002374 (+ 988us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 11:42:37.002741 (+ 367us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 11:42:37.002835 (+ 94us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 11:42:37.002843 (+ 8us) server_negotiation.cc:1035] Waiting for connection context
0620 11:42:37.003002 (+ 159us) server_negotiation.cc:299] Negotiation successful
0620 11:42:37.003175 (+ 173us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":163,"thread_start_us":91,"threads_started":1}
I20260620 11:42:37.003463 9244 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6100d64b207048afabe58fc85b390757" candidate_uuid: "170e23b744f94a62854cb7cd24ac7506" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "36f86b86532e42758e1f5249c186f526" is_pre_election: true
I20260620 11:42:37.003507 8964 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6100d64b207048afabe58fc85b390757" candidate_uuid: "170e23b744f94a62854cb7cd24ac7506" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "47814c4c6447409dbab77568f6434cbd" is_pre_election: true
I20260620 11:42:37.003605 8964 raft_consensus.cc:2454] T 6100d64b207048afabe58fc85b390757 P 47814c4c6447409dbab77568f6434cbd [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 170e23b744f94a62854cb7cd24ac7506 in term 0.
I20260620 11:42:37.003602 9244 raft_consensus.cc:2454] T 6100d64b207048afabe58fc85b390757 P 36f86b86532e42758e1f5249c186f526 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 170e23b744f94a62854cb7cd24ac7506 in term 0.
I20260620 11:42:37.003830 9037 leader_election.cc:302] T 6100d64b207048afabe58fc85b390757 P 170e23b744f94a62854cb7cd24ac7506 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 170e23b744f94a62854cb7cd24ac7506, 36f86b86532e42758e1f5249c186f526; no voters:
I20260620 11:42:37.004040 9395 raft_consensus.cc:2781] T 6100d64b207048afabe58fc85b390757 P 170e23b744f94a62854cb7cd24ac7506 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260620 11:42:37.004107 9395 raft_consensus.cc:482] T 6100d64b207048afabe58fc85b390757 P 170e23b744f94a62854cb7cd24ac7506 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260620 11:42:37.004125 9395 raft_consensus.cc:3037] T 6100d64b207048afabe58fc85b390757 P 170e23b744f94a62854cb7cd24ac7506 [term 0 FOLLOWER]: Advancing to term 1
I20260620 11:42:37.004812 9395 raft_consensus.cc:504] T 6100d64b207048afabe58fc85b390757 P 170e23b744f94a62854cb7cd24ac7506 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "47814c4c6447409dbab77568f6434cbd" member_type: VOTER last_known_addr { host: "127.6.234.193" port: 40543 } } peers { permanent_uuid: "170e23b744f94a62854cb7cd24ac7506" member_type: VOTER last_known_addr { host: "127.6.234.194" port: 44189 } } peers { permanent_uuid: "36f86b86532e42758e1f5249c186f526" member_type: VOTER last_known_addr { host: "127.6.234.195" port: 37197 } }
I20260620 11:42:37.004972 9395 leader_election.cc:288] T 6100d64b207048afabe58fc85b390757 P 170e23b744f94a62854cb7cd24ac7506 [CANDIDATE]: Term 1 election: Requested vote from peers 47814c4c6447409dbab77568f6434cbd (127.6.234.193:40543), 36f86b86532e42758e1f5249c186f526 (127.6.234.195:37197)
I20260620 11:42:37.005131 8964 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6100d64b207048afabe58fc85b390757" candidate_uuid: "170e23b744f94a62854cb7cd24ac7506" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "47814c4c6447409dbab77568f6434cbd"
I20260620 11:42:37.005213 8964 raft_consensus.cc:3037] T 6100d64b207048afabe58fc85b390757 P 47814c4c6447409dbab77568f6434cbd [term 0 FOLLOWER]: Advancing to term 1
I20260620 11:42:37.005759 8964 raft_consensus.cc:2454] T 6100d64b207048afabe58fc85b390757 P 47814c4c6447409dbab77568f6434cbd [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 170e23b744f94a62854cb7cd24ac7506 in term 1.
I20260620 11:42:37.005906 9244 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6100d64b207048afabe58fc85b390757" candidate_uuid: "170e23b744f94a62854cb7cd24ac7506" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "36f86b86532e42758e1f5249c186f526"
I20260620 11:42:37.005954 9035 leader_election.cc:302] T 6100d64b207048afabe58fc85b390757 P 170e23b744f94a62854cb7cd24ac7506 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 170e23b744f94a62854cb7cd24ac7506, 47814c4c6447409dbab77568f6434cbd; no voters:
I20260620 11:42:37.005985 9244 raft_consensus.cc:3037] T 6100d64b207048afabe58fc85b390757 P 36f86b86532e42758e1f5249c186f526 [term 0 FOLLOWER]: Advancing to term 1
I20260620 11:42:37.006059 9395 raft_consensus.cc:2781] T 6100d64b207048afabe58fc85b390757 P 170e23b744f94a62854cb7cd24ac7506 [term 1 FOLLOWER]: Leader election won for term 1
I20260620 11:42:37.006206 9395 raft_consensus.cc:686] T 6100d64b207048afabe58fc85b390757 P 170e23b744f94a62854cb7cd24ac7506 [term 1 LEADER]: Becoming Leader. State: Replica: 170e23b744f94a62854cb7cd24ac7506, State: Running, Role: LEADER
I20260620 11:42:37.006271 9395 consensus_queue.cc:237] T 6100d64b207048afabe58fc85b390757 P 170e23b744f94a62854cb7cd24ac7506 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "47814c4c6447409dbab77568f6434cbd" member_type: VOTER last_known_addr { host: "127.6.234.193" port: 40543 } } peers { permanent_uuid: "170e23b744f94a62854cb7cd24ac7506" member_type: VOTER last_known_addr { host: "127.6.234.194" port: 44189 } } peers { permanent_uuid: "36f86b86532e42758e1f5249c186f526" member_type: VOTER last_known_addr { host: "127.6.234.195" port: 37197 } }
I20260620 11:42:37.006778 9244 raft_consensus.cc:2454] T 6100d64b207048afabe58fc85b390757 P 36f86b86532e42758e1f5249c186f526 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 170e23b744f94a62854cb7cd24ac7506 in term 1.
I20260620 11:42:37.007285 8805 catalog_manager.cc:5495] T 6100d64b207048afabe58fc85b390757 P 170e23b744f94a62854cb7cd24ac7506 reported cstate change: term changed from 0 to 1, leader changed from <none> to 170e23b744f94a62854cb7cd24ac7506 (127.6.234.194). New cstate: current_term: 1 leader_uuid: "170e23b744f94a62854cb7cd24ac7506" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "47814c4c6447409dbab77568f6434cbd" member_type: VOTER last_known_addr { host: "127.6.234.193" port: 40543 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "170e23b744f94a62854cb7cd24ac7506" member_type: VOTER last_known_addr { host: "127.6.234.194" port: 44189 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "36f86b86532e42758e1f5249c186f526" member_type: VOTER last_known_addr { host: "127.6.234.195" port: 37197 } health_report { overall_health: UNKNOWN } } }
I20260620 11:42:37.026280 9404 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 11:42:37.024078 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53708 (local address 127.6.234.194:44189)
0620 11:42:37.024211 (+ 133us) server_negotiation.cc:206] Beginning negotiation
0620 11:42:37.024213 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 11:42:37.024221 (+ 8us) server_negotiation.cc:407] Connection header received
0620 11:42:37.024258 (+ 37us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 11:42:37.024258 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 11:42:37.024274 (+ 16us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 11:42:37.024342 (+ 68us) server_negotiation.cc:226] Negotiated authn=TOKEN
0620 11:42:37.024438 (+ 96us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 11:42:37.024879 (+ 441us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 11:42:37.025560 (+ 681us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 11:42:37.025827 (+ 267us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 11:42:37.025886 (+ 59us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 11:42:37.025999 (+ 113us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0620 11:42:37.026109 (+ 110us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0620 11:42:37.026161 (+ 52us) server_negotiation.cc:1035] Waiting for connection context
0620 11:42:37.026209 (+ 48us) server_negotiation.cc:299] Negotiation successful
0620 11:42:37.026222 (+ 13us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":111,"thread_start_us":83,"threads_started":1}
I20260620 11:42:37.031026 9400 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 11:42:37.028962 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.6.234.194:35665 (local address 127.6.234.195:37197)
0620 11:42:37.029024 (+ 62us) server_negotiation.cc:206] Beginning negotiation
0620 11:42:37.029026 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 11:42:37.029088 (+ 62us) server_negotiation.cc:407] Connection header received
0620 11:42:37.029137 (+ 49us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 11:42:37.029138 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 11:42:37.029145 (+ 7us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 11:42:37.029190 (+ 45us) server_negotiation.cc:226] Negotiated authn=TOKEN
0620 11:42:37.029285 (+ 95us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 11:42:37.029765 (+ 480us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 11:42:37.030469 (+ 704us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 11:42:37.030645 (+ 176us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 11:42:37.030691 (+ 46us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 11:42:37.030854 (+ 163us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0620 11:42:37.030876 (+ 22us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0620 11:42:37.030910 (+ 34us) server_negotiation.cc:1035] Waiting for connection context
0620 11:42:37.030962 (+ 52us) server_negotiation.cc:299] Negotiation successful
0620 11:42:37.030977 (+ 15us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":43}
I20260620 11:42:37.031039 9407 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 11:42:37.028834 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.234.195:37197 (local address 127.6.234.194:35665)
0620 11:42:37.029021 (+ 187us) negotiation.cc:107] Waiting for socket to connect
0620 11:42:37.029032 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0620 11:42:37.029097 (+ 65us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 11:42:37.029204 (+ 107us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 11:42:37.029204 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 11:42:37.029205 (+ 1us) client_negotiation.cc:189] Negotiated authn=TOKEN
0620 11:42:37.029247 (+ 42us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 11:42:37.029248 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 11:42:37.029838 (+ 590us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 11:42:37.029840 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 11:42:37.030397 (+ 557us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 11:42:37.030399 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 11:42:37.030714 (+ 315us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 11:42:37.030714 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 11:42:37.030769 (+ 55us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 11:42:37.030782 (+ 13us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0620 11:42:37.030924 (+ 142us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0620 11:42:37.030925 (+ 1us) client_negotiation.cc:769] Sending connection context
0620 11:42:37.030955 (+ 30us) client_negotiation.cc:240] Negotiation successful
0620 11:42:37.030985 (+ 30us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":147,"thread_start_us":83,"threads_started":1}
I20260620 11:42:37.032958 9244 raft_consensus.cc:1261] T 6100d64b207048afabe58fc85b390757 P 36f86b86532e42758e1f5249c186f526 [term 1 FOLLOWER]: Refusing update from remote peer 170e23b744f94a62854cb7cd24ac7506: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 11:42:37.033128 8964 raft_consensus.cc:1261] T 6100d64b207048afabe58fc85b390757 P 47814c4c6447409dbab77568f6434cbd [term 1 FOLLOWER]: Refusing update from remote peer 170e23b744f94a62854cb7cd24ac7506: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 11:42:37.033233 9401 consensus_queue.cc:1035] T 6100d64b207048afabe58fc85b390757 P 170e23b744f94a62854cb7cd24ac7506 [LEADER]: Connected to new peer: Peer: permanent_uuid: "36f86b86532e42758e1f5249c186f526" member_type: VOTER last_known_addr { host: "127.6.234.195" port: 37197 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 11:42:37.033327 9395 consensus_queue.cc:1035] T 6100d64b207048afabe58fc85b390757 P 170e23b744f94a62854cb7cd24ac7506 [LEADER]: Connected to new peer: Peer: permanent_uuid: "47814c4c6447409dbab77568f6434cbd" member_type: VOTER last_known_addr { host: "127.6.234.193" port: 40543 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 11:42:37.040113 9404 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 11:42:37.037944 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.6.234.195:34117 (local address 127.6.234.194:44189)
0620 11:42:37.038009 (+ 65us) server_negotiation.cc:206] Beginning negotiation
0620 11:42:37.038010 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0620 11:42:37.038105 (+ 95us) server_negotiation.cc:407] Connection header received
0620 11:42:37.038156 (+ 51us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 11:42:37.038157 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 11:42:37.038168 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 11:42:37.038221 (+ 53us) server_negotiation.cc:226] Negotiated authn=TOKEN
0620 11:42:37.038362 (+ 141us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 11:42:37.038855 (+ 493us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 11:42:37.039536 (+ 681us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 11:42:37.039703 (+ 167us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 11:42:37.039753 (+ 50us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 11:42:37.039904 (+ 151us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0620 11:42:37.039935 (+ 31us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0620 11:42:37.039984 (+ 49us) server_negotiation.cc:1035] Waiting for connection context
0620 11:42:37.040036 (+ 52us) server_negotiation.cc:299] Negotiation successful
0620 11:42:37.040054 (+ 18us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":43}
I20260620 11:42:37.040153 9411 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 11:42:37.037877 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.234.194:44189 (local address 127.6.234.195:34117)
0620 11:42:37.038030 (+ 153us) negotiation.cc:107] Waiting for socket to connect
0620 11:42:37.038040 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0620 11:42:37.038099 (+ 59us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 11:42:37.038231 (+ 132us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 11:42:37.038232 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 11:42:37.038233 (+ 1us) client_negotiation.cc:189] Negotiated authn=TOKEN
0620 11:42:37.038289 (+ 56us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 11:42:37.038291 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 11:42:37.038927 (+ 636us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 11:42:37.038927 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 11:42:37.039456 (+ 529us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 11:42:37.039458 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 11:42:37.039766 (+ 308us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 11:42:37.039766 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 11:42:37.039817 (+ 51us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 11:42:37.039820 (+ 3us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0620 11:42:37.039982 (+ 162us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0620 11:42:37.039984 (+ 2us) client_negotiation.cc:769] Sending connection context
0620 11:42:37.040034 (+ 50us) client_negotiation.cc:240] Negotiation successful
0620 11:42:37.040074 (+ 40us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":129,"thread_start_us":91,"threads_started":1}
I20260620 11:42:37.046777 8807 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:47686:
table { table_name: "test-table" } modify_external_catalogs: true
I20260620 11:42:37.046870 8807 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:47686:
table { table_name: "test-table" } modify_external_catalogs: true
I20260620 11:42:37.047497 8807 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 6e89f37d51dc404c95261d264710ef80: Sending DeleteTablet for 3 replicas of tablet 6100d64b207048afabe58fc85b390757
I20260620 11:42:37.047735 8944 tablet_service.cc:1514] Processing DeleteTablet for tablet 6100d64b207048afabe58fc85b390757 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 11:42:37 UTC) from {username='kudu', principal='kudu/127.6.234.254@KRBTEST.COM'} at 127.0.0.1:58340
I20260620 11:42:37.047751 9223 tablet_service.cc:1514] Processing DeleteTablet for tablet 6100d64b207048afabe58fc85b390757 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 11:42:37 UTC) from {username='kudu', principal='kudu/127.6.234.254@KRBTEST.COM'} at 127.0.0.1:45108
I20260620 11:42:37.047784 9084 tablet_service.cc:1514] Processing DeleteTablet for tablet 6100d64b207048afabe58fc85b390757 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 11:42:37 UTC) from {username='kudu', principal='kudu/127.6.234.254@KRBTEST.COM'} at 127.0.0.1:53692
I20260620 11:42:37.047850 9378 tablet_replica.cc:330] T 6100d64b207048afabe58fc85b390757 P 47814c4c6447409dbab77568f6434cbd: stopping tablet replica
I20260620 11:42:37.047883 9380 tablet_replica.cc:330] T 6100d64b207048afabe58fc85b390757 P 170e23b744f94a62854cb7cd24ac7506: stopping tablet replica
I20260620 11:42:37.047911 9378 raft_consensus.cc:2229] T 6100d64b207048afabe58fc85b390757 P 47814c4c6447409dbab77568f6434cbd [term 1 FOLLOWER]: Raft consensus shutting down.
I20260620 11:42:37.047923 9380 raft_consensus.cc:2229] T 6100d64b207048afabe58fc85b390757 P 170e23b744f94a62854cb7cd24ac7506 [term 1 LEADER]: Raft consensus shutting down.
I20260620 11:42:37.047956 9378 raft_consensus.cc:2258] T 6100d64b207048afabe58fc85b390757 P 47814c4c6447409dbab77568f6434cbd [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 11:42:37.047999 9380 raft_consensus.cc:2258] T 6100d64b207048afabe58fc85b390757 P 170e23b744f94a62854cb7cd24ac7506 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 11:42:37.048029 9379 tablet_replica.cc:330] T 6100d64b207048afabe58fc85b390757 P 36f86b86532e42758e1f5249c186f526: stopping tablet replica
I20260620 11:42:37.048081 9379 raft_consensus.cc:2229] T 6100d64b207048afabe58fc85b390757 P 36f86b86532e42758e1f5249c186f526 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260620 11:42:37.048116 9379 raft_consensus.cc:2258] T 6100d64b207048afabe58fc85b390757 P 36f86b86532e42758e1f5249c186f526 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 11:42:37.048295 9380 ts_tablet_manager.cc:1905] T 6100d64b207048afabe58fc85b390757 P 170e23b744f94a62854cb7cd24ac7506: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 11:42:37.048292 9378 ts_tablet_manager.cc:1905] T 6100d64b207048afabe58fc85b390757 P 47814c4c6447409dbab77568f6434cbd: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 11:42:37.048436 9379 ts_tablet_manager.cc:1905] T 6100d64b207048afabe58fc85b390757 P 36f86b86532e42758e1f5249c186f526: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 11:42:37.049947 9378 ts_tablet_manager.cc:1918] T 6100d64b207048afabe58fc85b390757 P 47814c4c6447409dbab77568f6434cbd: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260620 11:42:37.049975 9379 ts_tablet_manager.cc:1918] T 6100d64b207048afabe58fc85b390757 P 36f86b86532e42758e1f5249c186f526: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260620 11:42:37.049975 9380 ts_tablet_manager.cc:1918] T 6100d64b207048afabe58fc85b390757 P 170e23b744f94a62854cb7cd24ac7506: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260620 11:42:37.050006 9378 log.cc:1196] T 6100d64b207048afabe58fc85b390757 P 47814c4c6447409dbab77568f6434cbd: Deleting WAL directory at /tmp/dist-test-task1JRg7N/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781955746769051-7083-0/minicluster-data/ts-0/wal/wals/6100d64b207048afabe58fc85b390757
I20260620 11:42:37.050025 9380 log.cc:1196] T 6100d64b207048afabe58fc85b390757 P 170e23b744f94a62854cb7cd24ac7506: Deleting WAL directory at /tmp/dist-test-task1JRg7N/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781955746769051-7083-0/minicluster-data/ts-1/wal/wals/6100d64b207048afabe58fc85b390757
I20260620 11:42:37.050025 9379 log.cc:1196] T 6100d64b207048afabe58fc85b390757 P 36f86b86532e42758e1f5249c186f526: Deleting WAL directory at /tmp/dist-test-task1JRg7N/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781955746769051-7083-0/minicluster-data/ts-2/wal/wals/6100d64b207048afabe58fc85b390757
I20260620 11:42:37.050246 9378 ts_tablet_manager.cc:1939] T 6100d64b207048afabe58fc85b390757 P 47814c4c6447409dbab77568f6434cbd: Deleting consensus metadata
I20260620 11:42:37.050248 9379 ts_tablet_manager.cc:1939] T 6100d64b207048afabe58fc85b390757 P 36f86b86532e42758e1f5249c186f526: Deleting consensus metadata
I20260620 11:42:37.050248 9380 ts_tablet_manager.cc:1939] T 6100d64b207048afabe58fc85b390757 P 170e23b744f94a62854cb7cd24ac7506: Deleting consensus metadata
I20260620 11:42:37.050554 8791 catalog_manager.cc:4841] TS 47814c4c6447409dbab77568f6434cbd (127.6.234.193:40543): tablet 6100d64b207048afabe58fc85b390757 (table test-table [id=35a9ca682f3e45f397df98ddb835f55c]) successfully deleted
I20260620 11:42:37.050585 8793 catalog_manager.cc:4841] TS 170e23b744f94a62854cb7cd24ac7506 (127.6.234.194:44189): tablet 6100d64b207048afabe58fc85b390757 (table test-table [id=35a9ca682f3e45f397df98ddb835f55c]) successfully deleted
I20260620 11:42:37.050626 8793 catalog_manager.cc:4841] TS 36f86b86532e42758e1f5249c186f526 (127.6.234.195:37197): tablet 6100d64b207048afabe58fc85b390757 (table test-table [id=35a9ca682f3e45f397df98ddb835f55c]) successfully deleted
I20260620 11:42:37.050694 8807 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:47686:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260620 11:42:37.050846 8807 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260620 11:42:37.052153 9084 tablet_service.cc:1467] Processing CreateTablet for tablet 6404d5fc6e2e4a90834efbc0479c97bb (DEFAULT_TABLE table=test-table [id=a6f08e74ba7e4d3fb80f800317b926fc]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 11:42:37.052217 9223 tablet_service.cc:1467] Processing CreateTablet for tablet 6404d5fc6e2e4a90834efbc0479c97bb (DEFAULT_TABLE table=test-table [id=a6f08e74ba7e4d3fb80f800317b926fc]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 11:42:37.052268 9084 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 6404d5fc6e2e4a90834efbc0479c97bb. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 11:42:37.052263 8944 tablet_service.cc:1467] Processing CreateTablet for tablet 6404d5fc6e2e4a90834efbc0479c97bb (DEFAULT_TABLE table=test-table [id=a6f08e74ba7e4d3fb80f800317b926fc]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 11:42:37.052318 9223 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 6404d5fc6e2e4a90834efbc0479c97bb. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 11:42:37.052348 8944 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 6404d5fc6e2e4a90834efbc0479c97bb. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 11:42:37.053683 9390 tablet_bootstrap.cc:492] T 6404d5fc6e2e4a90834efbc0479c97bb P 47814c4c6447409dbab77568f6434cbd: Bootstrap starting.
I20260620 11:42:37.053711 9391 tablet_bootstrap.cc:492] T 6404d5fc6e2e4a90834efbc0479c97bb P 36f86b86532e42758e1f5249c186f526: Bootstrap starting.
I20260620 11:42:37.053689 9389 tablet_bootstrap.cc:492] T 6404d5fc6e2e4a90834efbc0479c97bb P 170e23b744f94a62854cb7cd24ac7506: Bootstrap starting.
I20260620 11:42:37.054052 9391 tablet_bootstrap.cc:654] T 6404d5fc6e2e4a90834efbc0479c97bb P 36f86b86532e42758e1f5249c186f526: Neither blocks nor log segments found. Creating new log.
I20260620 11:42:37.054118 9389 tablet_bootstrap.cc:654] T 6404d5fc6e2e4a90834efbc0479c97bb P 170e23b744f94a62854cb7cd24ac7506: Neither blocks nor log segments found. Creating new log.
I20260620 11:42:37.054404 9390 tablet_bootstrap.cc:654] T 6404d5fc6e2e4a90834efbc0479c97bb P 47814c4c6447409dbab77568f6434cbd: Neither blocks nor log segments found. Creating new log.
I20260620 11:42:37.054983 9389 tablet_bootstrap.cc:492] T 6404d5fc6e2e4a90834efbc0479c97bb P 170e23b744f94a62854cb7cd24ac7506: No bootstrap required, opened a new log
I20260620 11:42:37.055039 9389 ts_tablet_manager.cc:1397] T 6404d5fc6e2e4a90834efbc0479c97bb P 170e23b744f94a62854cb7cd24ac7506: Time spent bootstrapping tablet: real 0.001s user 0.001s sys 0.000s
I20260620 11:42:37.055212 9389 raft_consensus.cc:348] T 6404d5fc6e2e4a90834efbc0479c97bb P 170e23b744f94a62854cb7cd24ac7506 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "47814c4c6447409dbab77568f6434cbd" member_type: VOTER last_known_addr { host: "127.6.234.193" port: 40543 } } peers { permanent_uuid: "170e23b744f94a62854cb7cd24ac7506" member_type: VOTER last_known_addr { host: "127.6.234.194" port: 44189 } } peers { permanent_uuid: "36f86b86532e42758e1f5249c186f526" member_type: VOTER last_known_addr { host: "127.6.234.195" port: 37197 } }
I20260620 11:42:37.055287 9389 raft_consensus.cc:374] T 6404d5fc6e2e4a90834efbc0479c97bb P 170e23b744f94a62854cb7cd24ac7506 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 11:42:37.055308 9389 raft_consensus.cc:729] T 6404d5fc6e2e4a90834efbc0479c97bb P 170e23b744f94a62854cb7cd24ac7506 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 170e23b744f94a62854cb7cd24ac7506, State: Initialized, Role: FOLLOWER
I20260620 11:42:37.055332 9391 tablet_bootstrap.cc:492] T 6404d5fc6e2e4a90834efbc0479c97bb P 36f86b86532e42758e1f5249c186f526: No bootstrap required, opened a new log
I20260620 11:42:37.055349 9389 consensus_queue.cc:260] T 6404d5fc6e2e4a90834efbc0479c97bb P 170e23b744f94a62854cb7cd24ac7506 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "47814c4c6447409dbab77568f6434cbd" member_type: VOTER last_known_addr { host: "127.6.234.193" port: 40543 } } peers { permanent_uuid: "170e23b744f94a62854cb7cd24ac7506" member_type: VOTER last_known_addr { host: "127.6.234.194" port: 44189 } } peers { permanent_uuid: "36f86b86532e42758e1f5249c186f526" member_type: VOTER last_known_addr { host: "127.6.234.195" port: 37197 } }
I20260620 11:42:37.055382 9391 ts_tablet_manager.cc:1397] T 6404d5fc6e2e4a90834efbc0479c97bb P 36f86b86532e42758e1f5249c186f526: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260620 11:42:37.055471 9389 ts_tablet_manager.cc:1428] T 6404d5fc6e2e4a90834efbc0479c97bb P 170e23b744f94a62854cb7cd24ac7506: Time spent starting tablet: real 0.000s user 0.001s sys 0.000s
I20260620 11:42:37.055483 9391 raft_consensus.cc:348] T 6404d5fc6e2e4a90834efbc0479c97bb P 36f86b86532e42758e1f5249c186f526 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "47814c4c6447409dbab77568f6434cbd" member_type: VOTER last_known_addr { host: "127.6.234.193" port: 40543 } } peers { permanent_uuid: "170e23b744f94a62854cb7cd24ac7506" member_type: VOTER last_known_addr { host: "127.6.234.194" port: 44189 } } peers { permanent_uuid: "36f86b86532e42758e1f5249c186f526" member_type: VOTER last_known_addr { host: "127.6.234.195" port: 37197 } }
I20260620 11:42:37.055546 9391 raft_consensus.cc:374] T 6404d5fc6e2e4a90834efbc0479c97bb P 36f86b86532e42758e1f5249c186f526 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 11:42:37.055572 9391 raft_consensus.cc:729] T 6404d5fc6e2e4a90834efbc0479c97bb P 36f86b86532e42758e1f5249c186f526 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 36f86b86532e42758e1f5249c186f526, State: Initialized, Role: FOLLOWER
I20260620 11:42:37.055613 9391 consensus_queue.cc:260] T 6404d5fc6e2e4a90834efbc0479c97bb P 36f86b86532e42758e1f5249c186f526 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "47814c4c6447409dbab77568f6434cbd" member_type: VOTER last_known_addr { host: "127.6.234.193" port: 40543 } } peers { permanent_uuid: "170e23b744f94a62854cb7cd24ac7506" member_type: VOTER last_known_addr { host: "127.6.234.194" port: 44189 } } peers { permanent_uuid: "36f86b86532e42758e1f5249c186f526" member_type: VOTER last_known_addr { host: "127.6.234.195" port: 37197 } }
I20260620 11:42:37.055706 9391 ts_tablet_manager.cc:1428] T 6404d5fc6e2e4a90834efbc0479c97bb P 36f86b86532e42758e1f5249c186f526: Time spent starting tablet: real 0.000s user 0.001s sys 0.000s
I20260620 11:42:37.055814 9390 tablet_bootstrap.cc:492] T 6404d5fc6e2e4a90834efbc0479c97bb P 47814c4c6447409dbab77568f6434cbd: No bootstrap required, opened a new log
I20260620 11:42:37.055855 9390 ts_tablet_manager.cc:1397] T 6404d5fc6e2e4a90834efbc0479c97bb P 47814c4c6447409dbab77568f6434cbd: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260620 11:42:37.056003 9390 raft_consensus.cc:348] T 6404d5fc6e2e4a90834efbc0479c97bb P 47814c4c6447409dbab77568f6434cbd [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "47814c4c6447409dbab77568f6434cbd" member_type: VOTER last_known_addr { host: "127.6.234.193" port: 40543 } } peers { permanent_uuid: "170e23b744f94a62854cb7cd24ac7506" member_type: VOTER last_known_addr { host: "127.6.234.194" port: 44189 } } peers { permanent_uuid: "36f86b86532e42758e1f5249c186f526" member_type: VOTER last_known_addr { host: "127.6.234.195" port: 37197 } }
I20260620 11:42:37.056064 9390 raft_consensus.cc:374] T 6404d5fc6e2e4a90834efbc0479c97bb P 47814c4c6447409dbab77568f6434cbd [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 11:42:37.056084 9390 raft_consensus.cc:729] T 6404d5fc6e2e4a90834efbc0479c97bb P 47814c4c6447409dbab77568f6434cbd [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 47814c4c6447409dbab77568f6434cbd, State: Initialized, Role: FOLLOWER
I20260620 11:42:37.056124 9390 consensus_queue.cc:260] T 6404d5fc6e2e4a90834efbc0479c97bb P 47814c4c6447409dbab77568f6434cbd [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "47814c4c6447409dbab77568f6434cbd" member_type: VOTER last_known_addr { host: "127.6.234.193" port: 40543 } } peers { permanent_uuid: "170e23b744f94a62854cb7cd24ac7506" member_type: VOTER last_known_addr { host: "127.6.234.194" port: 44189 } } peers { permanent_uuid: "36f86b86532e42758e1f5249c186f526" member_type: VOTER last_known_addr { host: "127.6.234.195" port: 37197 } }
I20260620 11:42:37.056211 9390 ts_tablet_manager.cc:1428] T 6404d5fc6e2e4a90834efbc0479c97bb P 47814c4c6447409dbab77568f6434cbd: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
I20260620 11:42:37.083618 9396 raft_consensus.cc:482] T 6404d5fc6e2e4a90834efbc0479c97bb P 47814c4c6447409dbab77568f6434cbd [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260620 11:42:37.083714 9396 raft_consensus.cc:504] T 6404d5fc6e2e4a90834efbc0479c97bb P 47814c4c6447409dbab77568f6434cbd [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "47814c4c6447409dbab77568f6434cbd" member_type: VOTER last_known_addr { host: "127.6.234.193" port: 40543 } } peers { permanent_uuid: "170e23b744f94a62854cb7cd24ac7506" member_type: VOTER last_known_addr { host: "127.6.234.194" port: 44189 } } peers { permanent_uuid: "36f86b86532e42758e1f5249c186f526" member_type: VOTER last_known_addr { host: "127.6.234.195" port: 37197 } }
I20260620 11:42:37.084077 9396 leader_election.cc:288] T 6404d5fc6e2e4a90834efbc0479c97bb P 47814c4c6447409dbab77568f6434cbd [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 170e23b744f94a62854cb7cd24ac7506 (127.6.234.194:44189), 36f86b86532e42758e1f5249c186f526 (127.6.234.195:37197)
I20260620 11:42:37.087657 9417 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 11:42:37.084246 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.234.194:44189 (local address 127.6.234.193:56955)
0620 11:42:37.084389 (+ 143us) negotiation.cc:107] Waiting for socket to connect
0620 11:42:37.084401 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0620 11:42:37.084480 (+ 79us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 11:42:37.084716 (+ 236us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 11:42:37.084717 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 11:42:37.084720 (+ 3us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 11:42:37.084799 (+ 79us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 11:42:37.084801 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 11:42:37.085775 (+ 974us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 11:42:37.085776 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 11:42:37.086598 (+ 822us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 11:42:37.086600 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 11:42:37.087347 (+ 747us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 11:42:37.087348 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 11:42:37.087432 (+ 84us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 11:42:37.087437 (+ 5us) client_negotiation.cc:769] Sending connection context
0620 11:42:37.087499 (+ 62us) client_negotiation.cc:240] Negotiation successful
0620 11:42:37.087540 (+ 41us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":112,"mutex_wait_us":1,"thread_start_us":75,"threads_started":1}
I20260620 11:42:37.087661 9404 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 11:42:37.084320 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.6.234.193:56955 (local address 127.6.234.194:44189)
0620 11:42:37.084365 (+ 45us) server_negotiation.cc:206] Beginning negotiation
0620 11:42:37.084367 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 11:42:37.084477 (+ 110us) server_negotiation.cc:407] Connection header received
0620 11:42:37.084546 (+ 69us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 11:42:37.084547 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 11:42:37.084556 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 11:42:37.084684 (+ 128us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 11:42:37.084885 (+ 201us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 11:42:37.085649 (+ 764us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 11:42:37.086691 (+ 1042us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 11:42:37.087254 (+ 563us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 11:42:37.087333 (+ 79us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 11:42:37.087342 (+ 9us) server_negotiation.cc:1035] Waiting for connection context
0620 11:42:37.087515 (+ 173us) server_negotiation.cc:299] Negotiation successful
0620 11:42:37.087553 (+ 38us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":19}
I20260620 11:42:37.088099 9104 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6404d5fc6e2e4a90834efbc0479c97bb" candidate_uuid: "47814c4c6447409dbab77568f6434cbd" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "170e23b744f94a62854cb7cd24ac7506" is_pre_election: true
I20260620 11:42:37.088207 9104 raft_consensus.cc:2454] T 6404d5fc6e2e4a90834efbc0479c97bb P 170e23b744f94a62854cb7cd24ac7506 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 47814c4c6447409dbab77568f6434cbd in term 0.
I20260620 11:42:37.088464 8897 leader_election.cc:302] T 6404d5fc6e2e4a90834efbc0479c97bb P 47814c4c6447409dbab77568f6434cbd [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 170e23b744f94a62854cb7cd24ac7506, 47814c4c6447409dbab77568f6434cbd; no voters:
I20260620 11:42:37.088608 9396 raft_consensus.cc:2781] T 6404d5fc6e2e4a90834efbc0479c97bb P 47814c4c6447409dbab77568f6434cbd [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260620 11:42:37.088639 9396 raft_consensus.cc:482] T 6404d5fc6e2e4a90834efbc0479c97bb P 47814c4c6447409dbab77568f6434cbd [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260620 11:42:37.088658 9396 raft_consensus.cc:3037] T 6404d5fc6e2e4a90834efbc0479c97bb P 47814c4c6447409dbab77568f6434cbd [term 0 FOLLOWER]: Advancing to term 1
I20260620 11:42:37.088897 9418 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 11:42:37.084506 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.234.195:37197 (local address 127.6.234.193:48043)
0620 11:42:37.084638 (+ 132us) negotiation.cc:107] Waiting for socket to connect
0620 11:42:37.084648 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0620 11:42:37.084692 (+ 44us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 11:42:37.085771 (+ 1079us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 11:42:37.085772 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 11:42:37.085773 (+ 1us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 11:42:37.085831 (+ 58us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 11:42:37.085833 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 11:42:37.086758 (+ 925us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 11:42:37.086759 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 11:42:37.087829 (+ 1070us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 11:42:37.087832 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 11:42:37.088578 (+ 746us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 11:42:37.088579 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 11:42:37.088658 (+ 79us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 11:42:37.088663 (+ 5us) client_negotiation.cc:769] Sending connection context
0620 11:42:37.088769 (+ 106us) client_negotiation.cc:240] Negotiation successful
0620 11:42:37.088811 (+ 42us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":114,"thread_start_us":80,"threads_started":1}
I20260620 11:42:37.088917 9400 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 11:42:37.084627 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.6.234.193:48043 (local address 127.6.234.195:37197)
0620 11:42:37.084675 (+ 48us) server_negotiation.cc:206] Beginning negotiation
0620 11:42:37.084676 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0620 11:42:37.084685 (+ 9us) server_negotiation.cc:407] Connection header received
0620 11:42:37.084787 (+ 102us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 11:42:37.084788 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 11:42:37.084799 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 11:42:37.084856 (+ 57us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 11:42:37.085920 (+ 1064us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 11:42:37.086641 (+ 721us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 11:42:37.087922 (+ 1281us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 11:42:37.088496 (+ 574us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 11:42:37.088574 (+ 78us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 11:42:37.088582 (+ 8us) server_negotiation.cc:1035] Waiting for connection context
0620 11:42:37.088784 (+ 202us) server_negotiation.cc:299] Negotiation successful
0620 11:42:37.088836 (+ 52us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":22}
I20260620 11:42:37.089488 9396 raft_consensus.cc:504] T 6404d5fc6e2e4a90834efbc0479c97bb P 47814c4c6447409dbab77568f6434cbd [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "47814c4c6447409dbab77568f6434cbd" member_type: VOTER last_known_addr { host: "127.6.234.193" port: 40543 } } peers { permanent_uuid: "170e23b744f94a62854cb7cd24ac7506" member_type: VOTER last_known_addr { host: "127.6.234.194" port: 44189 } } peers { permanent_uuid: "36f86b86532e42758e1f5249c186f526" member_type: VOTER last_known_addr { host: "127.6.234.195" port: 37197 } }
I20260620 11:42:37.089566 9396 leader_election.cc:288] T 6404d5fc6e2e4a90834efbc0479c97bb P 47814c4c6447409dbab77568f6434cbd [CANDIDATE]: Term 1 election: Requested vote from peers 170e23b744f94a62854cb7cd24ac7506 (127.6.234.194:44189), 36f86b86532e42758e1f5249c186f526 (127.6.234.195:37197)
I20260620 11:42:37.089841 9104 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6404d5fc6e2e4a90834efbc0479c97bb" candidate_uuid: "47814c4c6447409dbab77568f6434cbd" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "170e23b744f94a62854cb7cd24ac7506"
I20260620 11:42:37.089911 9104 raft_consensus.cc:3037] T 6404d5fc6e2e4a90834efbc0479c97bb P 170e23b744f94a62854cb7cd24ac7506 [term 0 FOLLOWER]: Advancing to term 1
I20260620 11:42:37.090647 9104 raft_consensus.cc:2454] T 6404d5fc6e2e4a90834efbc0479c97bb P 170e23b744f94a62854cb7cd24ac7506 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 47814c4c6447409dbab77568f6434cbd in term 1.
I20260620 11:42:37.090808 8897 leader_election.cc:302] T 6404d5fc6e2e4a90834efbc0479c97bb P 47814c4c6447409dbab77568f6434cbd [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 170e23b744f94a62854cb7cd24ac7506, 47814c4c6447409dbab77568f6434cbd; no voters:
I20260620 11:42:37.090848 9244 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6404d5fc6e2e4a90834efbc0479c97bb" candidate_uuid: "47814c4c6447409dbab77568f6434cbd" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "36f86b86532e42758e1f5249c186f526" is_pre_election: true
I20260620 11:42:37.090855 9243 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6404d5fc6e2e4a90834efbc0479c97bb" candidate_uuid: "47814c4c6447409dbab77568f6434cbd" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "36f86b86532e42758e1f5249c186f526"
I20260620 11:42:37.090941 9244 raft_consensus.cc:2454] T 6404d5fc6e2e4a90834efbc0479c97bb P 36f86b86532e42758e1f5249c186f526 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 47814c4c6447409dbab77568f6434cbd in term 0.
I20260620 11:42:37.091045 9396 raft_consensus.cc:2781] T 6404d5fc6e2e4a90834efbc0479c97bb P 47814c4c6447409dbab77568f6434cbd [term 1 FOLLOWER]: Leader election won for term 1
I20260620 11:42:37.091190 9396 raft_consensus.cc:686] T 6404d5fc6e2e4a90834efbc0479c97bb P 47814c4c6447409dbab77568f6434cbd [term 1 LEADER]: Becoming Leader. State: Replica: 47814c4c6447409dbab77568f6434cbd, State: Running, Role: LEADER
I20260620 11:42:37.091262 9396 consensus_queue.cc:237] T 6404d5fc6e2e4a90834efbc0479c97bb P 47814c4c6447409dbab77568f6434cbd [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "47814c4c6447409dbab77568f6434cbd" member_type: VOTER last_known_addr { host: "127.6.234.193" port: 40543 } } peers { permanent_uuid: "170e23b744f94a62854cb7cd24ac7506" member_type: VOTER last_known_addr { host: "127.6.234.194" port: 44189 } } peers { permanent_uuid: "36f86b86532e42758e1f5249c186f526" member_type: VOTER last_known_addr { host: "127.6.234.195" port: 37197 } }
I20260620 11:42:37.092005 8805 catalog_manager.cc:5495] T 6404d5fc6e2e4a90834efbc0479c97bb P 47814c4c6447409dbab77568f6434cbd reported cstate change: term changed from 0 to 1, leader changed from <none> to 47814c4c6447409dbab77568f6434cbd (127.6.234.193). New cstate: current_term: 1 leader_uuid: "47814c4c6447409dbab77568f6434cbd" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "47814c4c6447409dbab77568f6434cbd" member_type: VOTER last_known_addr { host: "127.6.234.193" port: 40543 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "170e23b744f94a62854cb7cd24ac7506" member_type: VOTER last_known_addr { host: "127.6.234.194" port: 44189 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "36f86b86532e42758e1f5249c186f526" member_type: VOTER last_known_addr { host: "127.6.234.195" port: 37197 } health_report { overall_health: UNKNOWN } } }
I20260620 11:42:37.106878 9398 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 11:42:37.103528 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58348 (local address 127.6.234.193:40543)
0620 11:42:37.103562 (+ 34us) server_negotiation.cc:206] Beginning negotiation
0620 11:42:37.103564 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 11:42:37.103571 (+ 7us) server_negotiation.cc:407] Connection header received
0620 11:42:37.103600 (+ 29us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 11:42:37.103601 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 11:42:37.103619 (+ 18us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 11:42:37.103752 (+ 133us) server_negotiation.cc:226] Negotiated authn=TOKEN
0620 11:42:37.104233 (+ 481us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 11:42:37.104996 (+ 763us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 11:42:37.105831 (+ 835us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 11:42:37.106125 (+ 294us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 11:42:37.106183 (+ 58us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 11:42:37.106380 (+ 197us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0620 11:42:37.106522 (+ 142us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0620 11:42:37.106594 (+ 72us) server_negotiation.cc:1035] Waiting for connection context
0620 11:42:37.106775 (+ 181us) server_negotiation.cc:299] Negotiation successful
0620 11:42:37.106797 (+ 22us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":19}
I20260620 11:42:37.113140 9423 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 11:42:37.110417 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.234.195:37197 (local address 127.6.234.193:45619)
0620 11:42:37.110573 (+ 156us) negotiation.cc:107] Waiting for socket to connect
0620 11:42:37.110583 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0620 11:42:37.110639 (+ 56us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 11:42:37.110765 (+ 126us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 11:42:37.110766 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 11:42:37.110767 (+ 1us) client_negotiation.cc:189] Negotiated authn=TOKEN
0620 11:42:37.110810 (+ 43us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 11:42:37.110811 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 11:42:37.111655 (+ 844us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 11:42:37.111655 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 11:42:37.112340 (+ 685us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 11:42:37.112342 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 11:42:37.112789 (+ 447us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 11:42:37.112790 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 11:42:37.112832 (+ 42us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 11:42:37.112843 (+ 11us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0620 11:42:37.113016 (+ 173us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0620 11:42:37.113018 (+ 2us) client_negotiation.cc:769] Sending connection context
0620 11:42:37.113051 (+ 33us) client_negotiation.cc:240] Negotiation successful
0620 11:42:37.113082 (+ 31us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":124,"thread_start_us":87,"threads_started":1}
I20260620 11:42:37.113147 9400 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 11:42:37.110566 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.6.234.193:45619 (local address 127.6.234.195:37197)
0620 11:42:37.110603 (+ 37us) server_negotiation.cc:206] Beginning negotiation
0620 11:42:37.110605 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 11:42:37.110631 (+ 26us) server_negotiation.cc:407] Connection header received
0620 11:42:37.110696 (+ 65us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 11:42:37.110697 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 11:42:37.110706 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 11:42:37.110757 (+ 51us) server_negotiation.cc:226] Negotiated authn=TOKEN
0620 11:42:37.110866 (+ 109us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 11:42:37.111585 (+ 719us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 11:42:37.112422 (+ 837us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 11:42:37.112699 (+ 277us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 11:42:37.112771 (+ 72us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 11:42:37.112921 (+ 150us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0620 11:42:37.112956 (+ 35us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0620 11:42:37.113010 (+ 54us) server_negotiation.cc:1035] Waiting for connection context
0620 11:42:37.113054 (+ 44us) server_negotiation.cc:299] Negotiation successful
0620 11:42:37.113075 (+ 21us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":19}
I20260620 11:42:37.115340 9104 raft_consensus.cc:1261] T 6404d5fc6e2e4a90834efbc0479c97bb P 170e23b744f94a62854cb7cd24ac7506 [term 1 FOLLOWER]: Refusing update from remote peer 47814c4c6447409dbab77568f6434cbd: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 11:42:37.115463 9244 raft_consensus.cc:3037] T 6404d5fc6e2e4a90834efbc0479c97bb P 36f86b86532e42758e1f5249c186f526 [term 0 FOLLOWER]: Advancing to term 1
I20260620 11:42:37.116355 9244 raft_consensus.cc:1261] T 6404d5fc6e2e4a90834efbc0479c97bb P 36f86b86532e42758e1f5249c186f526 [term 1 FOLLOWER]: Refusing update from remote peer 47814c4c6447409dbab77568f6434cbd: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 11:42:37.116592 9419 consensus_queue.cc:1035] T 6404d5fc6e2e4a90834efbc0479c97bb P 47814c4c6447409dbab77568f6434cbd [LEADER]: Connected to new peer: Peer: permanent_uuid: "36f86b86532e42758e1f5249c186f526" member_type: VOTER last_known_addr { host: "127.6.234.195" port: 37197 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 11:42:37.116724 9396 consensus_queue.cc:1035] T 6404d5fc6e2e4a90834efbc0479c97bb P 47814c4c6447409dbab77568f6434cbd [LEADER]: Connected to new peer: Peer: permanent_uuid: "170e23b744f94a62854cb7cd24ac7506" member_type: VOTER last_known_addr { host: "127.6.234.194" port: 44189 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 11:42:37.123687 9398 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 11:42:37.121514 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.6.234.195:33829 (local address 127.6.234.193:40543)
0620 11:42:37.121550 (+ 36us) server_negotiation.cc:206] Beginning negotiation
0620 11:42:37.121552 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 11:42:37.121564 (+ 12us) server_negotiation.cc:407] Connection header received
0620 11:42:37.121648 (+ 84us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 11:42:37.121649 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 11:42:37.121659 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 11:42:37.121734 (+ 75us) server_negotiation.cc:226] Negotiated authn=TOKEN
0620 11:42:37.121871 (+ 137us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 11:42:37.122308 (+ 437us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 11:42:37.123030 (+ 722us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 11:42:37.123213 (+ 183us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 11:42:37.123282 (+ 69us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 11:42:37.123425 (+ 143us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0620 11:42:37.123459 (+ 34us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0620 11:42:37.123510 (+ 51us) server_negotiation.cc:1035] Waiting for connection context
0620 11:42:37.123599 (+ 89us) server_negotiation.cc:299] Negotiation successful
0620 11:42:37.123620 (+ 21us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":19}
I20260620 11:42:37.123692 9411 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 11:42:37.121484 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.234.193:40543 (local address 127.6.234.195:33829)
0620 11:42:37.121532 (+ 48us) negotiation.cc:107] Waiting for socket to connect
0620 11:42:37.121542 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0620 11:42:37.121579 (+ 37us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 11:42:37.121775 (+ 196us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 11:42:37.121776 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 11:42:37.121777 (+ 1us) client_negotiation.cc:189] Negotiated authn=TOKEN
0620 11:42:37.121817 (+ 40us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 11:42:37.121819 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 11:42:37.122391 (+ 572us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 11:42:37.122392 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 11:42:37.122927 (+ 535us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 11:42:37.122929 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 11:42:37.123292 (+ 363us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 11:42:37.123293 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 11:42:37.123345 (+ 52us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 11:42:37.123352 (+ 7us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0620 11:42:37.123524 (+ 172us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0620 11:42:37.123526 (+ 2us) client_negotiation.cc:769] Sending connection context
0620 11:42:37.123581 (+ 55us) client_negotiation.cc:240] Negotiation successful
0620 11:42:37.123627 (+ 46us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":29}
W20260620 11:42:37.124189 9363 tablet_replica.cc:1298] Aborted: operation has been aborted: cancelling pending write operations
I20260620 11:42:38.642475 8805 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:47686:
table { table_name: "test-table" } modify_external_catalogs: true
I20260620 11:42:38.642570 8805 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:47686:
table { table_name: "test-table" } modify_external_catalogs: true
I20260620 11:42:38.643360 8805 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 6e89f37d51dc404c95261d264710ef80: Sending DeleteTablet for 3 replicas of tablet 6404d5fc6e2e4a90834efbc0479c97bb
I20260620 11:42:38.643617 9084 tablet_service.cc:1514] Processing DeleteTablet for tablet 6404d5fc6e2e4a90834efbc0479c97bb with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 11:42:38 UTC) from {username='kudu', principal='kudu/127.6.234.254@KRBTEST.COM'} at 127.0.0.1:53692
I20260620 11:42:38.643625 8944 tablet_service.cc:1514] Processing DeleteTablet for tablet 6404d5fc6e2e4a90834efbc0479c97bb with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 11:42:38 UTC) from {username='kudu', principal='kudu/127.6.234.254@KRBTEST.COM'} at 127.0.0.1:58340
I20260620 11:42:38.643635 9223 tablet_service.cc:1514] Processing DeleteTablet for tablet 6404d5fc6e2e4a90834efbc0479c97bb with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 11:42:38 UTC) from {username='kudu', principal='kudu/127.6.234.254@KRBTEST.COM'} at 127.0.0.1:45108
I20260620 11:42:38.643865 9451 tablet_replica.cc:330] T 6404d5fc6e2e4a90834efbc0479c97bb P 47814c4c6447409dbab77568f6434cbd: stopping tablet replica
I20260620 11:42:38.643889 9452 tablet_replica.cc:330] T 6404d5fc6e2e4a90834efbc0479c97bb P 36f86b86532e42758e1f5249c186f526: stopping tablet replica
I20260620 11:42:38.643950 9452 raft_consensus.cc:2229] T 6404d5fc6e2e4a90834efbc0479c97bb P 36f86b86532e42758e1f5249c186f526 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260620 11:42:38.643981 7083 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task1JRg7N/build/release/bin/kudu with pid 8879
I20260620 11:42:38.644721 9453 tablet_replica.cc:330] T 6404d5fc6e2e4a90834efbc0479c97bb P 170e23b744f94a62854cb7cd24ac7506: stopping tablet replica
I20260620 11:42:38.644835 9453 raft_consensus.cc:2229] T 6404d5fc6e2e4a90834efbc0479c97bb P 170e23b744f94a62854cb7cd24ac7506 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260620 11:42:38.644948 9452 raft_consensus.cc:2258] T 6404d5fc6e2e4a90834efbc0479c97bb P 36f86b86532e42758e1f5249c186f526 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 11:42:38.645040 9453 raft_consensus.cc:2258] T 6404d5fc6e2e4a90834efbc0479c97bb P 170e23b744f94a62854cb7cd24ac7506 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 11:42:38.645295 9452 ts_tablet_manager.cc:1905] T 6404d5fc6e2e4a90834efbc0479c97bb P 36f86b86532e42758e1f5249c186f526: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 11:42:38.646718 9453 ts_tablet_manager.cc:1905] T 6404d5fc6e2e4a90834efbc0479c97bb P 170e23b744f94a62854cb7cd24ac7506: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 11:42:38.646986 9452 ts_tablet_manager.cc:1918] T 6404d5fc6e2e4a90834efbc0479c97bb P 36f86b86532e42758e1f5249c186f526: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.8
I20260620 11:42:38.647053 9452 log.cc:1196] T 6404d5fc6e2e4a90834efbc0479c97bb P 36f86b86532e42758e1f5249c186f526: Deleting WAL directory at /tmp/dist-test-task1JRg7N/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781955746769051-7083-0/minicluster-data/ts-2/wal/wals/6404d5fc6e2e4a90834efbc0479c97bb
I20260620 11:42:38.647291 9452 ts_tablet_manager.cc:1939] T 6404d5fc6e2e4a90834efbc0479c97bb P 36f86b86532e42758e1f5249c186f526: Deleting consensus metadata
I20260620 11:42:38.647635 8793 catalog_manager.cc:4841] TS 36f86b86532e42758e1f5249c186f526 (127.6.234.195:37197): tablet 6404d5fc6e2e4a90834efbc0479c97bb (table test-table [id=a6f08e74ba7e4d3fb80f800317b926fc]) successfully deleted
I20260620 11:42:38.648259 9453 ts_tablet_manager.cc:1918] T 6404d5fc6e2e4a90834efbc0479c97bb P 170e23b744f94a62854cb7cd24ac7506: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.8
I20260620 11:42:38.648339 9453 log.cc:1196] T 6404d5fc6e2e4a90834efbc0479c97bb P 170e23b744f94a62854cb7cd24ac7506: Deleting WAL directory at /tmp/dist-test-task1JRg7N/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781955746769051-7083-0/minicluster-data/ts-1/wal/wals/6404d5fc6e2e4a90834efbc0479c97bb
I20260620 11:42:38.648545 9453 ts_tablet_manager.cc:1939] T 6404d5fc6e2e4a90834efbc0479c97bb P 170e23b744f94a62854cb7cd24ac7506: Deleting consensus metadata
I20260620 11:42:38.648873 8793 catalog_manager.cc:4841] TS 170e23b744f94a62854cb7cd24ac7506 (127.6.234.194:44189): tablet 6404d5fc6e2e4a90834efbc0479c97bb (table test-table [id=a6f08e74ba7e4d3fb80f800317b926fc]) successfully deleted
W20260620 11:42:38.650888 8791 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.6.234.193:40543 (error 108)
I20260620 11:42:38.651064 7083 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task1JRg7N/build/release/bin/kudu with pid 9019
I20260620 11:42:38.652092 9455 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 11:42:38.651846 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.234.193:40543 (local address 127.0.0.1:58364)
0620 11:42:38.651970 (+ 124us) negotiation.cc:107] Waiting for socket to connect
0620 11:42:38.652057 (+ 87us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.234.193:40543: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":99,"thread_start_us":78,"threads_started":1}
W20260620 11:42:38.652249 8791 catalog_manager.cc:4584] TS 47814c4c6447409dbab77568f6434cbd (127.6.234.193:40543): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 6404d5fc6e2e4a90834efbc0479c97bb: Network error: Client connection negotiation failed: client connection to 127.6.234.193:40543: connect: Connection refused (error 111)
I20260620 11:42:38.657867 7083 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task1JRg7N/build/release/bin/kudu with pid 9159
I20260620 11:42:38.663897 7083 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task1JRg7N/build/release/bin/kudu with pid 8775
2026-06-20T11:42:38Z chronyd exiting
[ OK ] SecurityITest.TxnSmokeWithDifferentUserTypes (5644 ms)
[ RUN ] SecurityITest.TestNoKerberosCredentials
Loading random data
Initializing database '/tmp/dist-test-task1JRg7N/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781955746769051-7083-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 20 11:42:38 dist-test-slave-cjc6 krb5kdc[9458](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 20 11:42:38 dist-test-slave-cjc6 krb5kdc[9458](info): set up 2 sockets
Jun 20 11:42:38 dist-test-slave-cjc6 krb5kdc[9458](info): commencing operation
krb5kdc: starting...
W20260620 11:42:39.713476 7083 mini_kdc.cc:121] Time spent starting KDC: real 1.029s user 0.000s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task1JRg7N/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781955746769051-7083-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task1JRg7N/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781955746769051-7083-0/krb5kdc/test-user.keytab.
Jun 20 11:42:39 dist-test-slave-cjc6 krb5kdc[9458](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781955759, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-20T11:42:39Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-20T11:42:39Z Disabled control of system clock
WARNING: no policy specified for kudu/127.6.234.254@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.6.234.254@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.6.234.254 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task1JRg7N/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781955746769051-7083-0/krb5kdc/kudu_127.6.234.254.keytab.
Entry for principal kudu/127.6.234.254 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task1JRg7N/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781955746769051-7083-0/krb5kdc/kudu_127.6.234.254.keytab.
I20260620 11:42:39.826685 7083 external_mini_cluster.cc:1326] Running /tmp/dist-test-task1JRg7N/build/release/bin/kudu
/tmp/dist-test-task1JRg7N/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task1JRg7N/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781955746769051-7083-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task1JRg7N/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781955746769051-7083-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task1JRg7N/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781955746769051-7083-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task1JRg7N/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781955746769051-7083-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.6.234.254:39843
--webserver_interface=127.6.234.254
--webserver_port=0
--keytab_file=/tmp/dist-test-task1JRg7N/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781955746769051-7083-0/krb5kdc/kudu_127.6.234.254.keytab
--principal=kudu/127.6.234.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.6.234.212:35913
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.6.234.254:39843
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task1JRg7N/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781955746769051-7083-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task1JRg7N/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781955746769051-7083-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task1JRg7N/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781955746769051-7083-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 11:42:39.906803 9471 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 11:42:39.907002 9471 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 11:42:39.907025 9471 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 11:42:39.908192 9471 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 11:42:39.908236 9471 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 11:42:39.908250 9471 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 11:42:39.908264 9471 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 11:42:39.908277 9471 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 11:42:39.909765 9471 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.6.234.212:35913
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task1JRg7N/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781955746769051-7083-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task1JRg7N/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781955746769051-7083-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.6.234.254:39843
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task1JRg7N/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781955746769051-7083-0/krb5kdc/kudu_127.6.234.254.keytab
--principal=kudu/127.6.234.254
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.6.234.254:39843
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task1JRg7N/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781955746769051-7083-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.6.234.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.9471
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task1JRg7N/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781955746769051-7083-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 20 Jun 2026 10:32:37 UTC on ccycloud.quasar-bgyofv.root.comops.site
build id 4457173
I20260620 11:42:39.910012 9471 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 11:42:39.910255 9471 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260620 11:42:39.915112 9471 server_base.cc:1034] running on GCE node
W20260620 11:42:39.915087 9479 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 11:42:39.915086 9477 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 11:42:39.915120 9476 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 11:42:39.915562 9471 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 11:42:39.915792 9471 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 11:42:39.916932 9471 hybrid_clock.cc:648] HybridClock initialized: now 1781955759916911 us; error 36 us; skew 500 ppm
Jun 20 11:42:39 dist-test-slave-cjc6 krb5kdc[9458](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781955759, etypes {rep=17 tkt=17 ses=17}, kudu/127.6.234.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 11:42:39.919728 9471 init.cc:377] Logged in from keytab as kudu/127.6.234.254@KRBTEST.COM (short username kudu)
I20260620 11:42:39.920163 9471 webserver.cc:460] Webserver started at http://127.6.234.254:35357/ using document root <none> and password file <none>
I20260620 11:42:39.920342 9471 fs_manager.cc:359] Metadata directory not provided
I20260620 11:42:39.920380 9471 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 11:42:39.920475 9471 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 11:42:39.921500 9471 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task1JRg7N/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781955746769051-7083-0/minicluster-data/master-0/data/instance:
uuid: "aec03134440d4280973601ad0048daaf"
format_stamp: "Formatted at 2026-06-20 11:42:39 on dist-test-slave-cjc6"
server_key: "d633b803b11bbcc3f9800195c900c224"
server_key_iv: "3580c3655a3d2117105516d9ca9750c4"
server_key_version: "encryptionkey@0"
I20260620 11:42:39.921877 9471 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task1JRg7N/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781955746769051-7083-0/minicluster-data/master-0/wal/instance:
uuid: "aec03134440d4280973601ad0048daaf"
format_stamp: "Formatted at 2026-06-20 11:42:39 on dist-test-slave-cjc6"
server_key: "d633b803b11bbcc3f9800195c900c224"
server_key_iv: "3580c3655a3d2117105516d9ca9750c4"
server_key_version: "encryptionkey@0"
I20260620 11:42:39.923120 9471 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.001s
I20260620 11:42:39.923755 9486 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 11:42:39.923961 9471 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.001s
I20260620 11:42:39.924065 9471 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task1JRg7N/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781955746769051-7083-0/minicluster-data/master-0/data,/tmp/dist-test-task1JRg7N/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781955746769051-7083-0/minicluster-data/master-0/wal
uuid: "aec03134440d4280973601ad0048daaf"
format_stamp: "Formatted at 2026-06-20 11:42:39 on dist-test-slave-cjc6"
server_key: "d633b803b11bbcc3f9800195c900c224"
server_key_iv: "3580c3655a3d2117105516d9ca9750c4"
server_key_version: "encryptionkey@0"
I20260620 11:42:39.924144 9471 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task1JRg7N/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781955746769051-7083-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task1JRg7N/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781955746769051-7083-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task1JRg7N/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781955746769051-7083-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 11:42:39.958528 9471 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 11:42:39.960980 9471 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 11:42:39.961150 9471 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 11:42:39.965121 9471 rpc_server.cc:307] RPC server started. Bound to: 127.6.234.254:39843
I20260620 11:42:39.965157 9538 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.6.234.254:39843 every 8 connection(s)
I20260620 11:42:39.965541 9471 server_base.cc:1166] Dumped server information to /tmp/dist-test-task1JRg7N/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781955746769051-7083-0/minicluster-data/master-0/data/info.pb
I20260620 11:42:39.966427 9539 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 11:42:39.968638 9539 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P aec03134440d4280973601ad0048daaf: Bootstrap starting.
I20260620 11:42:39.969214 9539 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P aec03134440d4280973601ad0048daaf: Neither blocks nor log segments found. Creating new log.
I20260620 11:42:39.969513 9539 log.cc:824] T 00000000000000000000000000000000 P aec03134440d4280973601ad0048daaf: Log is configured to *not* fsync() on all Append() calls
I20260620 11:42:39.970184 9539 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P aec03134440d4280973601ad0048daaf: No bootstrap required, opened a new log
I20260620 11:42:39.971447 7083 external_mini_cluster.cc:1388] Started /tmp/dist-test-task1JRg7N/build/release/bin/kudu as pid 9471
I20260620 11:42:39.971552 7083 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task1JRg7N/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781955746769051-7083-0/minicluster-data/master-0/wal/instance
I20260620 11:42:39.971725 7083 external_mini_cluster.cc:1428] Setting key fc1992299b3196e9d3aa2bbfe32ae80e
I20260620 11:42:39.972117 9539 raft_consensus.cc:348] T 00000000000000000000000000000000 P aec03134440d4280973601ad0048daaf [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "aec03134440d4280973601ad0048daaf" member_type: VOTER last_known_addr { host: "127.6.234.254" port: 39843 } }
I20260620 11:42:39.972321 9539 raft_consensus.cc:374] T 00000000000000000000000000000000 P aec03134440d4280973601ad0048daaf [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 11:42:39.972363 9539 raft_consensus.cc:729] T 00000000000000000000000000000000 P aec03134440d4280973601ad0048daaf [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: aec03134440d4280973601ad0048daaf, State: Initialized, Role: FOLLOWER
I20260620 11:42:39.972536 9539 consensus_queue.cc:260] T 00000000000000000000000000000000 P aec03134440d4280973601ad0048daaf [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "aec03134440d4280973601ad0048daaf" member_type: VOTER last_known_addr { host: "127.6.234.254" port: 39843 } }
I20260620 11:42:39.972604 9539 raft_consensus.cc:388] T 00000000000000000000000000000000 P aec03134440d4280973601ad0048daaf [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260620 11:42:39.972625 9539 raft_consensus.cc:482] T 00000000000000000000000000000000 P aec03134440d4280973601ad0048daaf [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260620 11:42:39.972692 9539 raft_consensus.cc:3037] T 00000000000000000000000000000000 P aec03134440d4280973601ad0048daaf [term 0 FOLLOWER]: Advancing to term 1
I20260620 11:42:39.973536 9539 raft_consensus.cc:504] T 00000000000000000000000000000000 P aec03134440d4280973601ad0048daaf [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "aec03134440d4280973601ad0048daaf" member_type: VOTER last_known_addr { host: "127.6.234.254" port: 39843 } }
I20260620 11:42:39.973623 9539 leader_election.cc:302] T 00000000000000000000000000000000 P aec03134440d4280973601ad0048daaf [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: aec03134440d4280973601ad0048daaf; no voters:
I20260620 11:42:39.973842 9539 leader_election.cc:288] T 00000000000000000000000000000000 P aec03134440d4280973601ad0048daaf [CANDIDATE]: Term 1 election: Requested vote from peers
I20260620 11:42:39.973869 9544 raft_consensus.cc:2781] T 00000000000000000000000000000000 P aec03134440d4280973601ad0048daaf [term 1 FOLLOWER]: Leader election won for term 1
I20260620 11:42:39.973975 9544 raft_consensus.cc:686] T 00000000000000000000000000000000 P aec03134440d4280973601ad0048daaf [term 1 LEADER]: Becoming Leader. State: Replica: aec03134440d4280973601ad0048daaf, State: Running, Role: LEADER
I20260620 11:42:39.974063 9544 consensus_queue.cc:237] T 00000000000000000000000000000000 P aec03134440d4280973601ad0048daaf [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "aec03134440d4280973601ad0048daaf" member_type: VOTER last_known_addr { host: "127.6.234.254" port: 39843 } }
I20260620 11:42:39.974120 9539 sys_catalog.cc:564] T 00000000000000000000000000000000 P aec03134440d4280973601ad0048daaf [sys.catalog]: configured and running, proceeding with master startup.
I20260620 11:42:39.974370 9544 sys_catalog.cc:455] T 00000000000000000000000000000000 P aec03134440d4280973601ad0048daaf [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "aec03134440d4280973601ad0048daaf" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "aec03134440d4280973601ad0048daaf" member_type: VOTER last_known_addr { host: "127.6.234.254" port: 39843 } } }
I20260620 11:42:39.974542 9544 sys_catalog.cc:458] T 00000000000000000000000000000000 P aec03134440d4280973601ad0048daaf [sys.catalog]: This master's current role is: LEADER
I20260620 11:42:39.974738 9544 sys_catalog.cc:455] T 00000000000000000000000000000000 P aec03134440d4280973601ad0048daaf [sys.catalog]: SysCatalogTable state changed. Reason: New leader aec03134440d4280973601ad0048daaf. Latest consensus state: current_term: 1 leader_uuid: "aec03134440d4280973601ad0048daaf" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "aec03134440d4280973601ad0048daaf" member_type: VOTER last_known_addr { host: "127.6.234.254" port: 39843 } } }
I20260620 11:42:39.974809 9544 sys_catalog.cc:458] T 00000000000000000000000000000000 P aec03134440d4280973601ad0048daaf [sys.catalog]: This master's current role is: LEADER
I20260620 11:42:39.975057 9552 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260620 11:42:39.975554 9552 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260620 11:42:39.977226 9552 catalog_manager.cc:1347] Generated new cluster ID: 32177c5f5afc443e8636a1a9eb27e86c
I20260620 11:42:39.977275 9552 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
Jun 20 11:42:39 dist-test-slave-cjc6 krb5kdc[9458](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781955759, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.6.234.254@KRBTEST.COM
I20260620 11:42:39.983163 9543 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 11:42:39.972242 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39686 (local address 127.6.234.254:39843)
0620 11:42:39.972439 (+ 197us) server_negotiation.cc:206] Beginning negotiation
0620 11:42:39.972449 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0620 11:42:39.972481 (+ 32us) server_negotiation.cc:407] Connection header received
0620 11:42:39.972597 (+ 116us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 11:42:39.972606 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 11:42:39.972664 (+ 58us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 11:42:39.972808 (+ 144us) server_negotiation.cc:226] Negotiated authn=SASL
0620 11:42:39.973427 (+ 619us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 11:42:39.974845 (+ 1418us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 11:42:39.975981 (+ 1136us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 11:42:39.976469 (+ 488us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 11:42:39.976597 (+ 128us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 11:42:39.979369 (+ 2772us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 11:42:39.979373 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 11:42:39.979374 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 11:42:39.979384 (+ 10us) server_negotiation.cc:922] calling sasl_server_start()
0620 11:42:39.981259 (+ 1875us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 11:42:39.981615 (+ 356us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 11:42:39.981618 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 11:42:39.981618 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 11:42:39.981758 (+ 140us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 11:42:39.981930 (+ 172us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 11:42:39.981931 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 11:42:39.981931 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 11:42:39.982204 (+ 273us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 11:42:39.982352 (+ 148us) server_negotiation.cc:1035] Waiting for connection context
0620 11:42:39.982836 (+ 484us) server_negotiation.cc:299] Negotiation successful
0620 11:42:39.982963 (+ 127us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":153,"thread_start_us":101,"threads_started":1}
I20260620 11:42:39.990901 9552 catalog_manager.cc:1370] Generated new certificate authority record
I20260620 11:42:39.991611 9552 catalog_manager.cc:1504] Loading token signing keys...
I20260620 11:42:40.001875 9552 catalog_manager.cc:5868] T 00000000000000000000000000000000 P aec03134440d4280973601ad0048daaf: Generated new TSK 0
I20260620 11:42:40.002125 9552 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.6.234.193@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.6.234.193@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.6.234.193 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task1JRg7N/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781955746769051-7083-0/krb5kdc/kudu_127.6.234.193.keytab.
Entry for principal kudu/127.6.234.193 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task1JRg7N/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781955746769051-7083-0/krb5kdc/kudu_127.6.234.193.keytab.
I20260620 11:42:40.029049 7083 external_mini_cluster.cc:1326] Running /tmp/dist-test-task1JRg7N/build/release/bin/kudu
/tmp/dist-test-task1JRg7N/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task1JRg7N/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781955746769051-7083-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-task1JRg7N/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781955746769051-7083-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task1JRg7N/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781955746769051-7083-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-task1JRg7N/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781955746769051-7083-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.6.234.193:0
--local_ip_for_outbound_sockets=127.6.234.193
--webserver_interface=127.6.234.193
--webserver_port=0
--tserver_master_addrs=127.6.234.254:39843
--keytab_file=/tmp/dist-test-task1JRg7N/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781955746769051-7083-0/krb5kdc/kudu_127.6.234.193.keytab
--principal=kudu/127.6.234.193
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.6.234.212:35913
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task1JRg7N/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781955746769051-7083-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task1JRg7N/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781955746769051-7083-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task1JRg7N/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781955746769051-7083-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 11:42:40.110769 9565 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 11:42:40.110970 9565 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 11:42:40.111001 9565 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 11:42:40.112082 9565 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 11:42:40.112116 9565 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 11:42:40.112142 9565 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.6.234.193
I20260620 11:42:40.113561 9565 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.6.234.212:35913
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task1JRg7N/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781955746769051-7083-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-task1JRg7N/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781955746769051-7083-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task1JRg7N/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781955746769051-7083-0/krb5kdc/kudu_127.6.234.193.keytab
--principal=kudu/127.6.234.193
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.6.234.193:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task1JRg7N/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781955746769051-7083-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.6.234.193
--webserver_port=0
--tserver_master_addrs=127.6.234.254:39843
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.9565
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.6.234.193
--log_dir=/tmp/dist-test-task1JRg7N/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781955746769051-7083-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 20 Jun 2026 10:32:37 UTC on ccycloud.quasar-bgyofv.root.comops.site
build id 4457173
I20260620 11:42:40.113811 9565 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 11:42:40.113988 9565 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 11:42:40.118800 9571 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 11:42:40.118798 9570 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 11:42:40.118892 9573 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 11:42:40.118960 9565 server_base.cc:1034] running on GCE node
I20260620 11:42:40.119338 9565 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 11:42:40.119537 9565 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 11:42:40.120702 9565 hybrid_clock.cc:648] HybridClock initialized: now 1781955760120687 us; error 26 us; skew 500 ppm
Jun 20 11:42:40 dist-test-slave-cjc6 krb5kdc[9458](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781955760, etypes {rep=17 tkt=17 ses=17}, kudu/127.6.234.193@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 11:42:40.123191 9565 init.cc:377] Logged in from keytab as kudu/127.6.234.193@KRBTEST.COM (short username kudu)
I20260620 11:42:40.123656 9565 webserver.cc:460] Webserver started at http://127.6.234.193:38939/ using document root <none> and password file <none>
I20260620 11:42:40.123886 9565 fs_manager.cc:359] Metadata directory not provided
I20260620 11:42:40.123952 9565 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 11:42:40.124111 9565 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 11:42:40.125118 9565 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task1JRg7N/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781955746769051-7083-0/minicluster-data/ts-0/data/instance:
uuid: "b7d27024b50a441f8742abbc36c3c4a1"
format_stamp: "Formatted at 2026-06-20 11:42:40 on dist-test-slave-cjc6"
server_key: "1edaf2f2bfb540aca309aa87d84a3419"
server_key_iv: "63056117d2e66821cbe35d6f19e2aeb4"
server_key_version: "encryptionkey@0"
I20260620 11:42:40.125474 9565 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task1JRg7N/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781955746769051-7083-0/minicluster-data/ts-0/wal/instance:
uuid: "b7d27024b50a441f8742abbc36c3c4a1"
format_stamp: "Formatted at 2026-06-20 11:42:40 on dist-test-slave-cjc6"
server_key: "1edaf2f2bfb540aca309aa87d84a3419"
server_key_iv: "63056117d2e66821cbe35d6f19e2aeb4"
server_key_version: "encryptionkey@0"
I20260620 11:42:40.126865 9565 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.001s sys 0.002s
I20260620 11:42:40.127636 9580 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 11:42:40.127827 9565 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260620 11:42:40.127933 9565 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task1JRg7N/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781955746769051-7083-0/minicluster-data/ts-0/data,/tmp/dist-test-task1JRg7N/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781955746769051-7083-0/minicluster-data/ts-0/wal
uuid: "b7d27024b50a441f8742abbc36c3c4a1"
format_stamp: "Formatted at 2026-06-20 11:42:40 on dist-test-slave-cjc6"
server_key: "1edaf2f2bfb540aca309aa87d84a3419"
server_key_iv: "63056117d2e66821cbe35d6f19e2aeb4"
server_key_version: "encryptionkey@0"
I20260620 11:42:40.128031 9565 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task1JRg7N/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781955746769051-7083-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-task1JRg7N/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781955746769051-7083-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-task1JRg7N/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781955746769051-7083-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 11:42:40.151444 9565 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 11:42:40.153399 9565 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 11:42:40.153569 9565 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 11:42:40.153887 9565 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 11:42:40.154214 9565 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 11:42:40.154270 9565 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 11:42:40.154310 9565 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 11:42:40.154338 9565 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 11:42:40.160118 9565 rpc_server.cc:307] RPC server started. Bound to: 127.6.234.193:42189
I20260620 11:42:40.160166 9693 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.6.234.193:42189 every 8 connection(s)
I20260620 11:42:40.160509 9565 server_base.cc:1166] Dumped server information to /tmp/dist-test-task1JRg7N/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781955746769051-7083-0/minicluster-data/ts-0/data/info.pb
I20260620 11:42:40.164012 7083 external_mini_cluster.cc:1388] Started /tmp/dist-test-task1JRg7N/build/release/bin/kudu as pid 9565
I20260620 11:42:40.164090 7083 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task1JRg7N/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781955746769051-7083-0/minicluster-data/ts-0/wal/instance
I20260620 11:42:40.164264 7083 external_mini_cluster.cc:1428] Setting key 34f0d8d8959f6a86892380adf2601e33
Jun 20 11:42:40 dist-test-slave-cjc6 krb5kdc[9458](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781955760, etypes {rep=17 tkt=17 ses=17}, kudu/127.6.234.193@KRBTEST.COM for kudu/127.6.234.254@KRBTEST.COM
I20260620 11:42:40.170195 9543 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 11:42:40.161281 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.6.234.193:41231 (local address 127.6.234.254:39843)
0620 11:42:40.161353 (+ 72us) server_negotiation.cc:206] Beginning negotiation
0620 11:42:40.161355 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 11:42:40.161556 (+ 201us) server_negotiation.cc:407] Connection header received
0620 11:42:40.162121 (+ 565us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 11:42:40.162122 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 11:42:40.162133 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 11:42:40.162192 (+ 59us) server_negotiation.cc:226] Negotiated authn=SASL
0620 11:42:40.163117 (+ 925us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 11:42:40.163607 (+ 490us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 11:42:40.164782 (+ 1175us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 11:42:40.165035 (+ 253us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 11:42:40.165109 (+ 74us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 11:42:40.167230 (+ 2121us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 11:42:40.167231 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 11:42:40.167231 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 11:42:40.167240 (+ 9us) server_negotiation.cc:922] calling sasl_server_start()
0620 11:42:40.168670 (+ 1430us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 11:42:40.169176 (+ 506us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 11:42:40.169177 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 11:42:40.169178 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 11:42:40.169225 (+ 47us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 11:42:40.169522 (+ 297us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 11:42:40.169522 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 11:42:40.169523 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 11:42:40.169619 (+ 96us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 11:42:40.169669 (+ 50us) server_negotiation.cc:1035] Waiting for connection context
0620 11:42:40.170062 (+ 393us) server_negotiation.cc:299] Negotiation successful
0620 11:42:40.170132 (+ 70us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":39}
I20260620 11:42:40.170495 9696 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 11:42:40.161265 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.234.254:39843 (local address 127.6.234.193:41231)
0620 11:42:40.161431 (+ 166us) negotiation.cc:107] Waiting for socket to connect
0620 11:42:40.161471 (+ 40us) client_negotiation.cc:174] Beginning negotiation
0620 11:42:40.162032 (+ 561us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 11:42:40.162262 (+ 230us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 11:42:40.162265 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 11:42:40.162835 (+ 570us) client_negotiation.cc:189] Negotiated authn=SASL
0620 11:42:40.163036 (+ 201us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 11:42:40.163040 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 11:42:40.163683 (+ 643us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 11:42:40.163684 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 11:42:40.164709 (+ 1025us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 11:42:40.164711 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 11:42:40.165131 (+ 420us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 11:42:40.165132 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 11:42:40.165241 (+ 109us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 11:42:40.165788 (+ 547us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 11:42:40.165798 (+ 10us) client_negotiation.cc:656] calling sasl_client_start()
0620 11:42:40.167127 (+ 1329us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 11:42:40.168758 (+ 1631us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 11:42:40.168765 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 11:42:40.168767 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0620 11:42:40.169100 (+ 333us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 11:42:40.169291 (+ 191us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 11:42:40.169291 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 11:42:40.169292 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0620 11:42:40.169451 (+ 159us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 11:42:40.169688 (+ 237us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 11:42:40.169717 (+ 29us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 11:42:40.169902 (+ 185us) client_negotiation.cc:769] Sending connection context
0620 11:42:40.170031 (+ 129us) client_negotiation.cc:240] Negotiation successful
0620 11:42:40.170216 (+ 185us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":126,"thread_start_us":95,"threads_started":1}
I20260620 11:42:40.171063 9694 heartbeater.cc:347] Connected to a master server at 127.6.234.254:39843
I20260620 11:42:40.171168 9694 heartbeater.cc:464] Registering TS with master...
I20260620 11:42:40.171371 9694 heartbeater.cc:511] Master 127.6.234.254:39843 requested a full tablet report, sending...
I20260620 11:42:40.171972 9503 ts_manager.cc:194] Registered new tserver with Master: b7d27024b50a441f8742abbc36c3c4a1 (127.6.234.193:42189)
I20260620 11:42:40.172601 9503 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.6.234.193@KRBTEST.COM'} at 127.6.234.193:41231
WARNING: no policy specified for kudu/127.6.234.194@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.6.234.194@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.6.234.194 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task1JRg7N/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781955746769051-7083-0/krb5kdc/kudu_127.6.234.194.keytab.
Entry for principal kudu/127.6.234.194 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task1JRg7N/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781955746769051-7083-0/krb5kdc/kudu_127.6.234.194.keytab.
I20260620 11:42:40.188426 7083 external_mini_cluster.cc:1326] Running /tmp/dist-test-task1JRg7N/build/release/bin/kudu
/tmp/dist-test-task1JRg7N/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task1JRg7N/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781955746769051-7083-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-task1JRg7N/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781955746769051-7083-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task1JRg7N/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781955746769051-7083-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-task1JRg7N/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781955746769051-7083-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.6.234.194:0
--local_ip_for_outbound_sockets=127.6.234.194
--webserver_interface=127.6.234.194
--webserver_port=0
--tserver_master_addrs=127.6.234.254:39843
--keytab_file=/tmp/dist-test-task1JRg7N/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781955746769051-7083-0/krb5kdc/kudu_127.6.234.194.keytab
--principal=kudu/127.6.234.194
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.6.234.212:35913
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task1JRg7N/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781955746769051-7083-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task1JRg7N/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781955746769051-7083-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task1JRg7N/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781955746769051-7083-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 11:42:40.267583 9699 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 11:42:40.267786 9699 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 11:42:40.267808 9699 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 11:42:40.268976 9699 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 11:42:40.269021 9699 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 11:42:40.269048 9699 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.6.234.194
I20260620 11:42:40.270499 9699 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.6.234.212:35913
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task1JRg7N/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781955746769051-7083-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-task1JRg7N/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781955746769051-7083-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task1JRg7N/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781955746769051-7083-0/krb5kdc/kudu_127.6.234.194.keytab
--principal=kudu/127.6.234.194
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.6.234.194:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task1JRg7N/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781955746769051-7083-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.6.234.194
--webserver_port=0
--tserver_master_addrs=127.6.234.254:39843
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.9699
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.6.234.194
--log_dir=/tmp/dist-test-task1JRg7N/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781955746769051-7083-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 20 Jun 2026 10:32:37 UTC on ccycloud.quasar-bgyofv.root.comops.site
build id 4457173
I20260620 11:42:40.270746 9699 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 11:42:40.270998 9699 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 11:42:40.276173 9705 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 11:42:40.276191 9704 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 11:42:40.276203 9707 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 11:42:40.276546 9699 server_base.cc:1034] running on GCE node
I20260620 11:42:40.276741 9699 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 11:42:40.276978 9699 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 11:42:40.278141 9699 hybrid_clock.cc:648] HybridClock initialized: now 1781955760278115 us; error 36 us; skew 500 ppm
Jun 20 11:42:40 dist-test-slave-cjc6 krb5kdc[9458](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781955760, etypes {rep=17 tkt=17 ses=17}, kudu/127.6.234.194@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 11:42:40.280616 9699 init.cc:377] Logged in from keytab as kudu/127.6.234.194@KRBTEST.COM (short username kudu)
I20260620 11:42:40.281044 9699 webserver.cc:460] Webserver started at http://127.6.234.194:44681/ using document root <none> and password file <none>
I20260620 11:42:40.281280 9699 fs_manager.cc:359] Metadata directory not provided
I20260620 11:42:40.281358 9699 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 11:42:40.281507 9699 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 11:42:40.282634 9699 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task1JRg7N/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781955746769051-7083-0/minicluster-data/ts-1/data/instance:
uuid: "9e80081073ac49c6acbc85b77ac942cf"
format_stamp: "Formatted at 2026-06-20 11:42:40 on dist-test-slave-cjc6"
server_key: "32551468aa23e0f7bd9f947d3de7227d"
server_key_iv: "0eb8c3465c504dd732d60347544d36eb"
server_key_version: "encryptionkey@0"
I20260620 11:42:40.283012 9699 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task1JRg7N/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781955746769051-7083-0/minicluster-data/ts-1/wal/instance:
uuid: "9e80081073ac49c6acbc85b77ac942cf"
format_stamp: "Formatted at 2026-06-20 11:42:40 on dist-test-slave-cjc6"
server_key: "32551468aa23e0f7bd9f947d3de7227d"
server_key_iv: "0eb8c3465c504dd732d60347544d36eb"
server_key_version: "encryptionkey@0"
I20260620 11:42:40.284294 9699 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260620 11:42:40.285030 9714 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 11:42:40.285238 9699 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260620 11:42:40.285347 9699 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task1JRg7N/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781955746769051-7083-0/minicluster-data/ts-1/data,/tmp/dist-test-task1JRg7N/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781955746769051-7083-0/minicluster-data/ts-1/wal
uuid: "9e80081073ac49c6acbc85b77ac942cf"
format_stamp: "Formatted at 2026-06-20 11:42:40 on dist-test-slave-cjc6"
server_key: "32551468aa23e0f7bd9f947d3de7227d"
server_key_iv: "0eb8c3465c504dd732d60347544d36eb"
server_key_version: "encryptionkey@0"
I20260620 11:42:40.285432 9699 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task1JRg7N/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781955746769051-7083-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-task1JRg7N/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781955746769051-7083-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-task1JRg7N/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781955746769051-7083-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 11:42:40.298000 9699 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 11:42:40.300124 9699 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 11:42:40.300304 9699 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 11:42:40.300565 9699 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 11:42:40.300886 9699 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 11:42:40.300942 9699 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 11:42:40.300987 9699 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 11:42:40.301012 9699 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 11:42:40.306949 9699 rpc_server.cc:307] RPC server started. Bound to: 127.6.234.194:41945
I20260620 11:42:40.307026 9827 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.6.234.194:41945 every 8 connection(s)
I20260620 11:42:40.307363 9699 server_base.cc:1166] Dumped server information to /tmp/dist-test-task1JRg7N/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781955746769051-7083-0/minicluster-data/ts-1/data/info.pb
I20260620 11:42:40.313019 7083 external_mini_cluster.cc:1388] Started /tmp/dist-test-task1JRg7N/build/release/bin/kudu as pid 9699
I20260620 11:42:40.313105 7083 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task1JRg7N/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781955746769051-7083-0/minicluster-data/ts-1/wal/instance
I20260620 11:42:40.313330 7083 external_mini_cluster.cc:1428] Setting key 187f3e428009cadd97b5be5717cd0857
Jun 20 11:42:40 dist-test-slave-cjc6 krb5kdc[9458](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781955760, etypes {rep=17 tkt=17 ses=17}, kudu/127.6.234.194@KRBTEST.COM for kudu/127.6.234.254@KRBTEST.COM
I20260620 11:42:40.317314 9830 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 11:42:40.308068 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.234.254:39843 (local address 127.6.234.194:35131)
0620 11:42:40.308266 (+ 198us) negotiation.cc:107] Waiting for socket to connect
0620 11:42:40.308327 (+ 61us) client_negotiation.cc:174] Beginning negotiation
0620 11:42:40.308884 (+ 557us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 11:42:40.309152 (+ 268us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 11:42:40.309157 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 11:42:40.309826 (+ 669us) client_negotiation.cc:189] Negotiated authn=SASL
0620 11:42:40.310029 (+ 203us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 11:42:40.310033 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 11:42:40.310754 (+ 721us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 11:42:40.310755 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 11:42:40.311560 (+ 805us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 11:42:40.311562 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 11:42:40.311985 (+ 423us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 11:42:40.311986 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 11:42:40.312109 (+ 123us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 11:42:40.312578 (+ 469us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 11:42:40.312585 (+ 7us) client_negotiation.cc:656] calling sasl_client_start()
0620 11:42:40.314246 (+ 1661us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 11:42:40.315908 (+ 1662us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 11:42:40.315916 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 11:42:40.315918 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0620 11:42:40.316153 (+ 235us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 11:42:40.316377 (+ 224us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 11:42:40.316378 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 11:42:40.316379 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0620 11:42:40.316497 (+ 118us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 11:42:40.316783 (+ 286us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 11:42:40.316788 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 11:42:40.316932 (+ 144us) client_negotiation.cc:769] Sending connection context
0620 11:42:40.317030 (+ 98us) client_negotiation.cc:240] Negotiation successful
0620 11:42:40.317152 (+ 122us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":151,"thread_start_us":92,"threads_started":1}
I20260620 11:42:40.317319 9543 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 11:42:40.308103 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.6.234.194:35131 (local address 127.6.234.254:39843)
0620 11:42:40.308202 (+ 99us) server_negotiation.cc:206] Beginning negotiation
0620 11:42:40.308206 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 11:42:40.308428 (+ 222us) server_negotiation.cc:407] Connection header received
0620 11:42:40.309002 (+ 574us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 11:42:40.309003 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 11:42:40.309015 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 11:42:40.309076 (+ 61us) server_negotiation.cc:226] Negotiated authn=SASL
0620 11:42:40.310133 (+ 1057us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 11:42:40.310651 (+ 518us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 11:42:40.311664 (+ 1013us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 11:42:40.311885 (+ 221us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 11:42:40.311976 (+ 91us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 11:42:40.314342 (+ 2366us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 11:42:40.314344 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 11:42:40.314345 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 11:42:40.314353 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0620 11:42:40.315763 (+ 1410us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 11:42:40.316246 (+ 483us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 11:42:40.316248 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 11:42:40.316248 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 11:42:40.316312 (+ 64us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 11:42:40.316568 (+ 256us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 11:42:40.316569 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 11:42:40.316569 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 11:42:40.316714 (+ 145us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 11:42:40.316790 (+ 76us) server_negotiation.cc:1035] Waiting for connection context
0620 11:42:40.317076 (+ 286us) server_negotiation.cc:299] Negotiation successful
0620 11:42:40.317176 (+ 100us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":45}
I20260620 11:42:40.317842 9828 heartbeater.cc:347] Connected to a master server at 127.6.234.254:39843
I20260620 11:42:40.317972 9828 heartbeater.cc:464] Registering TS with master...
I20260620 11:42:40.318154 9828 heartbeater.cc:511] Master 127.6.234.254:39843 requested a full tablet report, sending...
I20260620 11:42:40.318554 9503 ts_manager.cc:194] Registered new tserver with Master: 9e80081073ac49c6acbc85b77ac942cf (127.6.234.194:41945)
I20260620 11:42:40.319132 9503 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.6.234.194@KRBTEST.COM'} at 127.6.234.194:35131
WARNING: no policy specified for kudu/127.6.234.195@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.6.234.195@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.6.234.195 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task1JRg7N/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781955746769051-7083-0/krb5kdc/kudu_127.6.234.195.keytab.
Entry for principal kudu/127.6.234.195 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task1JRg7N/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781955746769051-7083-0/krb5kdc/kudu_127.6.234.195.keytab.
I20260620 11:42:40.337953 7083 external_mini_cluster.cc:1326] Running /tmp/dist-test-task1JRg7N/build/release/bin/kudu
/tmp/dist-test-task1JRg7N/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task1JRg7N/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781955746769051-7083-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-task1JRg7N/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781955746769051-7083-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task1JRg7N/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781955746769051-7083-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-task1JRg7N/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781955746769051-7083-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.6.234.195:0
--local_ip_for_outbound_sockets=127.6.234.195
--webserver_interface=127.6.234.195
--webserver_port=0
--tserver_master_addrs=127.6.234.254:39843
--keytab_file=/tmp/dist-test-task1JRg7N/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781955746769051-7083-0/krb5kdc/kudu_127.6.234.195.keytab
--principal=kudu/127.6.234.195
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.6.234.212:35913
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task1JRg7N/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781955746769051-7083-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task1JRg7N/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781955746769051-7083-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task1JRg7N/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781955746769051-7083-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 11:42:40.418124 9833 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 11:42:40.418337 9833 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 11:42:40.418360 9833 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 11:42:40.419529 9833 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 11:42:40.419597 9833 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 11:42:40.419637 9833 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.6.234.195
I20260620 11:42:40.421026 9833 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.6.234.212:35913
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task1JRg7N/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781955746769051-7083-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-task1JRg7N/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781955746769051-7083-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task1JRg7N/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781955746769051-7083-0/krb5kdc/kudu_127.6.234.195.keytab
--principal=kudu/127.6.234.195
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.6.234.195:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task1JRg7N/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781955746769051-7083-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.6.234.195
--webserver_port=0
--tserver_master_addrs=127.6.234.254:39843
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.9833
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.6.234.195
--log_dir=/tmp/dist-test-task1JRg7N/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781955746769051-7083-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 20 Jun 2026 10:32:37 UTC on ccycloud.quasar-bgyofv.root.comops.site
build id 4457173
I20260620 11:42:40.421267 9833 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 11:42:40.421500 9833 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260620 11:42:40.426659 9833 server_base.cc:1034] running on GCE node
W20260620 11:42:40.426599 9838 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 11:42:40.426581 9841 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 11:42:40.426551 9839 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 11:42:40.427086 9833 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 11:42:40.427350 9833 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 11:42:40.428540 9833 hybrid_clock.cc:648] HybridClock initialized: now 1781955760428526 us; error 25 us; skew 500 ppm
Jun 20 11:42:40 dist-test-slave-cjc6 krb5kdc[9458](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781955760, etypes {rep=17 tkt=17 ses=17}, kudu/127.6.234.195@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 11:42:40.431243 9833 init.cc:377] Logged in from keytab as kudu/127.6.234.195@KRBTEST.COM (short username kudu)
I20260620 11:42:40.431746 9833 webserver.cc:460] Webserver started at http://127.6.234.195:36617/ using document root <none> and password file <none>
I20260620 11:42:40.431988 9833 fs_manager.cc:359] Metadata directory not provided
I20260620 11:42:40.432058 9833 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 11:42:40.432196 9833 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 11:42:40.433223 9833 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task1JRg7N/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781955746769051-7083-0/minicluster-data/ts-2/data/instance:
uuid: "41bcf6f9e01943d88d450d29e950d57e"
format_stamp: "Formatted at 2026-06-20 11:42:40 on dist-test-slave-cjc6"
server_key: "5ae0718c6e03638bd1ff61f24487f4c7"
server_key_iv: "625cdbd9e082f3b918f013e28a26b753"
server_key_version: "encryptionkey@0"
I20260620 11:42:40.433601 9833 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task1JRg7N/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781955746769051-7083-0/minicluster-data/ts-2/wal/instance:
uuid: "41bcf6f9e01943d88d450d29e950d57e"
format_stamp: "Formatted at 2026-06-20 11:42:40 on dist-test-slave-cjc6"
server_key: "5ae0718c6e03638bd1ff61f24487f4c7"
server_key_iv: "625cdbd9e082f3b918f013e28a26b753"
server_key_version: "encryptionkey@0"
I20260620 11:42:40.435222 9833 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.001s
I20260620 11:42:40.436035 9848 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 11:42:40.436254 9833 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260620 11:42:40.436370 9833 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task1JRg7N/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781955746769051-7083-0/minicluster-data/ts-2/data,/tmp/dist-test-task1JRg7N/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781955746769051-7083-0/minicluster-data/ts-2/wal
uuid: "41bcf6f9e01943d88d450d29e950d57e"
format_stamp: "Formatted at 2026-06-20 11:42:40 on dist-test-slave-cjc6"
server_key: "5ae0718c6e03638bd1ff61f24487f4c7"
server_key_iv: "625cdbd9e082f3b918f013e28a26b753"
server_key_version: "encryptionkey@0"
I20260620 11:42:40.436462 9833 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task1JRg7N/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781955746769051-7083-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-task1JRg7N/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781955746769051-7083-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-task1JRg7N/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781955746769051-7083-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 11:42:40.457062 9833 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 11:42:40.459086 9833 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 11:42:40.459262 9833 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 11:42:40.459517 9833 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 11:42:40.459837 9833 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 11:42:40.459892 9833 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 11:42:40.459933 9833 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 11:42:40.459977 9833 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 11:42:40.465973 9833 rpc_server.cc:307] RPC server started. Bound to: 127.6.234.195:33407
I20260620 11:42:40.466017 9961 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.6.234.195:33407 every 8 connection(s)
I20260620 11:42:40.466349 9833 server_base.cc:1166] Dumped server information to /tmp/dist-test-task1JRg7N/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781955746769051-7083-0/minicluster-data/ts-2/data/info.pb
Jun 20 11:42:40 dist-test-slave-cjc6 krb5kdc[9458](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781955760, etypes {rep=17 tkt=17 ses=17}, kudu/127.6.234.195@KRBTEST.COM for kudu/127.6.234.254@KRBTEST.COM
I20260620 11:42:40.472832 7083 external_mini_cluster.cc:1388] Started /tmp/dist-test-task1JRg7N/build/release/bin/kudu as pid 9833
I20260620 11:42:40.472942 7083 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task1JRg7N/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781955746769051-7083-0/minicluster-data/ts-2/wal/instance
I20260620 11:42:40.473145 7083 external_mini_cluster.cc:1428] Setting key 70ca5ba6442949a1fbd54bd86eaddeed
I20260620 11:42:40.476342 9543 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 11:42:40.467212 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.6.234.195:37951 (local address 127.6.234.254:39843)
0620 11:42:40.467261 (+ 49us) server_negotiation.cc:206] Beginning negotiation
0620 11:42:40.467262 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0620 11:42:40.467500 (+ 238us) server_negotiation.cc:407] Connection header received
0620 11:42:40.468048 (+ 548us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 11:42:40.468049 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 11:42:40.468062 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 11:42:40.468119 (+ 57us) server_negotiation.cc:226] Negotiated authn=SASL
0620 11:42:40.469096 (+ 977us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 11:42:40.469609 (+ 513us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 11:42:40.470658 (+ 1049us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 11:42:40.470873 (+ 215us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 11:42:40.470963 (+ 90us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 11:42:40.473219 (+ 2256us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 11:42:40.473221 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 11:42:40.473222 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 11:42:40.473229 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0620 11:42:40.474872 (+ 1643us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 11:42:40.475319 (+ 447us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 11:42:40.475320 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 11:42:40.475320 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 11:42:40.475392 (+ 72us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 11:42:40.475634 (+ 242us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 11:42:40.475635 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 11:42:40.475635 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 11:42:40.475787 (+ 152us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 11:42:40.475842 (+ 55us) server_negotiation.cc:1035] Waiting for connection context
0620 11:42:40.476195 (+ 353us) server_negotiation.cc:299] Negotiation successful
0620 11:42:40.476278 (+ 83us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":23}
I20260620 11:42:40.476471 9964 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 11:42:40.467179 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.234.254:39843 (local address 127.6.234.195:37951)
0620 11:42:40.467374 (+ 195us) negotiation.cc:107] Waiting for socket to connect
0620 11:42:40.467411 (+ 37us) client_negotiation.cc:174] Beginning negotiation
0620 11:42:40.467936 (+ 525us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 11:42:40.468194 (+ 258us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 11:42:40.468198 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 11:42:40.468794 (+ 596us) client_negotiation.cc:189] Negotiated authn=SASL
0620 11:42:40.468987 (+ 193us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 11:42:40.468991 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 11:42:40.469734 (+ 743us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 11:42:40.469735 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 11:42:40.470561 (+ 826us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 11:42:40.470563 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 11:42:40.470972 (+ 409us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 11:42:40.470973 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 11:42:40.471103 (+ 130us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 11:42:40.471560 (+ 457us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 11:42:40.471568 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0620 11:42:40.473108 (+ 1540us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 11:42:40.474959 (+ 1851us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 11:42:40.474965 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 11:42:40.474966 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0620 11:42:40.475220 (+ 254us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 11:42:40.475472 (+ 252us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 11:42:40.475473 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 11:42:40.475474 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0620 11:42:40.475571 (+ 97us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 11:42:40.475865 (+ 294us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 11:42:40.475871 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 11:42:40.476029 (+ 158us) client_negotiation.cc:769] Sending connection context
0620 11:42:40.476138 (+ 109us) client_negotiation.cc:240] Negotiation successful
0620 11:42:40.476260 (+ 122us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":158,"thread_start_us":100,"threads_started":1}
I20260620 11:42:40.476941 9962 heartbeater.cc:347] Connected to a master server at 127.6.234.254:39843
I20260620 11:42:40.477037 9962 heartbeater.cc:464] Registering TS with master...
I20260620 11:42:40.477176 9962 heartbeater.cc:511] Master 127.6.234.254:39843 requested a full tablet report, sending...
I20260620 11:42:40.477541 9503 ts_manager.cc:194] Registered new tserver with Master: 41bcf6f9e01943d88d450d29e950d57e (127.6.234.195:33407)
I20260620 11:42:40.477965 9503 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.6.234.195@KRBTEST.COM'} at 127.6.234.195:37951
I20260620 11:42:40.485006 7083 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
W20260620 11:42:40.496111 9543 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 11:42:40.495172 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39702 (local address 127.6.234.254:39843)
0620 11:42:40.495251 (+ 79us) server_negotiation.cc:206] Beginning negotiation
0620 11:42:40.495253 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 11:42:40.495358 (+ 105us) server_negotiation.cc:407] Connection header received
0620 11:42:40.495407 (+ 49us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 11:42:40.495408 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 11:42:40.495417 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 11:42:40.495477 (+ 60us) server_negotiation.cc:226] Negotiated authn=SASL
0620 11:42:40.496032 (+ 555us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:39702: BlockingRecv error: recv got EOF from 127.0.0.1:39702 (error 108)
Metrics: {"server-negotiator.queue_time_us":46}
I20260620 11:42:40.497375 7083 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task1JRg7N/build/release/bin/kudu with pid 9565
I20260620 11:42:40.501941 7083 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task1JRg7N/build/release/bin/kudu with pid 9699
I20260620 11:42:40.505949 7083 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task1JRg7N/build/release/bin/kudu with pid 9833
I20260620 11:42:40.510087 7083 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task1JRg7N/build/release/bin/kudu with pid 9471
2026-06-20T11:42:40Z chronyd exiting
[ OK ] SecurityITest.TestNoKerberosCredentials (1845 ms)
[ RUN ] SecurityITest.SaslPlainFallback
Loading random data
Initializing database '/tmp/dist-test-task1JRg7N/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1781955746769051-7083-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 20 11:42:40 dist-test-slave-cjc6 krb5kdc[9974](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 20 11:42:40 dist-test-slave-cjc6 krb5kdc[9974](info): set up 2 sockets
Jun 20 11:42:40 dist-test-slave-cjc6 krb5kdc[9974](info): commencing operation
krb5kdc: starting...
W20260620 11:42:41.549549 7083 mini_kdc.cc:121] Time spent starting KDC: real 1.019s user 0.003s sys 0.002s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task1JRg7N/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1781955746769051-7083-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task1JRg7N/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1781955746769051-7083-0/krb5kdc/test-user.keytab.
Jun 20 11:42:41 dist-test-slave-cjc6 krb5kdc[9974](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781955761, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-20T11:42:41Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-20T11:42:41Z Disabled control of system clock
WARNING: no policy specified for kudu/127.6.234.254@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.6.234.254@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.6.234.254 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task1JRg7N/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1781955746769051-7083-0/krb5kdc/kudu_127.6.234.254.keytab.
Entry for principal kudu/127.6.234.254 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task1JRg7N/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1781955746769051-7083-0/krb5kdc/kudu_127.6.234.254.keytab.
I20260620 11:42:41.661089 7083 external_mini_cluster.cc:1326] Running /tmp/dist-test-task1JRg7N/build/release/bin/kudu
/tmp/dist-test-task1JRg7N/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task1JRg7N/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1781955746769051-7083-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task1JRg7N/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1781955746769051-7083-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task1JRg7N/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1781955746769051-7083-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task1JRg7N/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1781955746769051-7083-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.6.234.254:44391
--webserver_interface=127.6.234.254
--webserver_port=0
--keytab_file=/tmp/dist-test-task1JRg7N/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1781955746769051-7083-0/krb5kdc/kudu_127.6.234.254.keytab
--principal=kudu/127.6.234.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.6.234.212:34731
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.6.234.254:44391
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--rpc-authentication=optional
--user-acl=* with env {KRB5CCNAME=/tmp/dist-test-task1JRg7N/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1781955746769051-7083-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task1JRg7N/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1781955746769051-7083-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task1JRg7N/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1781955746769051-7083-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 11:42:41.742129 9987 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 11:42:41.742344 9987 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 11:42:41.742383 9987 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 11:42:41.743485 9987 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 11:42:41.743551 9987 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 11:42:41.743575 9987 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 11:42:41.743607 9987 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 11:42:41.743633 9987 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 11:42:41.745101 9987 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.6.234.212:34731
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task1JRg7N/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1781955746769051-7083-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task1JRg7N/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1781955746769051-7083-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.6.234.254:44391
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task1JRg7N/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1781955746769051-7083-0/krb5kdc/kudu_127.6.234.254.keytab
--principal=kudu/127.6.234.254
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.6.234.254:44391
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task1JRg7N/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1781955746769051-7083-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.6.234.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.9987
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task1JRg7N/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1781955746769051-7083-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 20 Jun 2026 10:32:37 UTC on ccycloud.quasar-bgyofv.root.comops.site
build id 4457173
I20260620 11:42:41.745362 9987 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 11:42:41.745620 9987 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260620 11:42:41.750294 9987 server_base.cc:1034] running on GCE node
W20260620 11:42:41.750233 9992 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 11:42:41.750270 9995 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 11:42:41.750238 9993 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 11:42:41.750689 9987 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 11:42:41.750926 9987 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 11:42:41.752075 9987 hybrid_clock.cc:648] HybridClock initialized: now 1781955761752065 us; error 30 us; skew 500 ppm
Jun 20 11:42:41 dist-test-slave-cjc6 krb5kdc[9974](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781955761, etypes {rep=17 tkt=17 ses=17}, kudu/127.6.234.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 11:42:41.754601 9987 init.cc:377] Logged in from keytab as kudu/127.6.234.254@KRBTEST.COM (short username kudu)
I20260620 11:42:41.755082 9987 webserver.cc:460] Webserver started at http://127.6.234.254:44497/ using document root <none> and password file <none>
I20260620 11:42:41.755311 9987 fs_manager.cc:359] Metadata directory not provided
I20260620 11:42:41.755378 9987 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 11:42:41.755524 9987 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 11:42:41.756543 9987 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task1JRg7N/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1781955746769051-7083-0/minicluster-data/master-0/data/instance:
uuid: "3c89e4953e5c441faff1541676a57476"
format_stamp: "Formatted at 2026-06-20 11:42:41 on dist-test-slave-cjc6"
server_key: "5a8f70fc0ea5c6aa05b4f62048573896"
server_key_iv: "fb9dab5146548fde3ae044765cc4cd66"
server_key_version: "encryptionkey@0"
I20260620 11:42:41.756899 9987 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task1JRg7N/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1781955746769051-7083-0/minicluster-data/master-0/wal/instance:
uuid: "3c89e4953e5c441faff1541676a57476"
format_stamp: "Formatted at 2026-06-20 11:42:41 on dist-test-slave-cjc6"
server_key: "5a8f70fc0ea5c6aa05b4f62048573896"
server_key_iv: "fb9dab5146548fde3ae044765cc4cd66"
server_key_version: "encryptionkey@0"
I20260620 11:42:41.758260 9987 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.001s sys 0.002s
I20260620 11:42:41.758940 10002 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 11:42:41.759106 9987 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260620 11:42:41.759210 9987 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task1JRg7N/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1781955746769051-7083-0/minicluster-data/master-0/data,/tmp/dist-test-task1JRg7N/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1781955746769051-7083-0/minicluster-data/master-0/wal
uuid: "3c89e4953e5c441faff1541676a57476"
format_stamp: "Formatted at 2026-06-20 11:42:41 on dist-test-slave-cjc6"
server_key: "5a8f70fc0ea5c6aa05b4f62048573896"
server_key_iv: "fb9dab5146548fde3ae044765cc4cd66"
server_key_version: "encryptionkey@0"
I20260620 11:42:41.759282 9987 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task1JRg7N/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1781955746769051-7083-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task1JRg7N/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1781955746769051-7083-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task1JRg7N/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1781955746769051-7083-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 11:42:41.774699 9987 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 11:42:41.776943 9987 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 11:42:41.777132 9987 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 11:42:41.781131 9987 rpc_server.cc:307] RPC server started. Bound to: 127.6.234.254:44391
I20260620 11:42:41.781167 10054 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.6.234.254:44391 every 8 connection(s)
I20260620 11:42:41.781503 9987 server_base.cc:1166] Dumped server information to /tmp/dist-test-task1JRg7N/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1781955746769051-7083-0/minicluster-data/master-0/data/info.pb
I20260620 11:42:41.782380 10055 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 11:42:41.784557 10055 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 3c89e4953e5c441faff1541676a57476: Bootstrap starting.
I20260620 11:42:41.785086 10055 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 3c89e4953e5c441faff1541676a57476: Neither blocks nor log segments found. Creating new log.
I20260620 11:42:41.785327 10055 log.cc:824] T 00000000000000000000000000000000 P 3c89e4953e5c441faff1541676a57476: Log is configured to *not* fsync() on all Append() calls
I20260620 11:42:41.786031 10055 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 3c89e4953e5c441faff1541676a57476: No bootstrap required, opened a new log
I20260620 11:42:41.786187 7083 external_mini_cluster.cc:1388] Started /tmp/dist-test-task1JRg7N/build/release/bin/kudu as pid 9987
I20260620 11:42:41.786295 7083 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task1JRg7N/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1781955746769051-7083-0/minicluster-data/master-0/wal/instance
I20260620 11:42:41.786468 7083 external_mini_cluster.cc:1428] Setting key 70a55ad6248fec802f9edc0a627d12bc
I20260620 11:42:41.788651 10055 raft_consensus.cc:348] T 00000000000000000000000000000000 P 3c89e4953e5c441faff1541676a57476 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3c89e4953e5c441faff1541676a57476" member_type: VOTER last_known_addr { host: "127.6.234.254" port: 44391 } }
I20260620 11:42:41.788836 10055 raft_consensus.cc:374] T 00000000000000000000000000000000 P 3c89e4953e5c441faff1541676a57476 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 11:42:41.788926 10055 raft_consensus.cc:729] T 00000000000000000000000000000000 P 3c89e4953e5c441faff1541676a57476 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3c89e4953e5c441faff1541676a57476, State: Initialized, Role: FOLLOWER
I20260620 11:42:41.789083 10055 consensus_queue.cc:260] T 00000000000000000000000000000000 P 3c89e4953e5c441faff1541676a57476 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3c89e4953e5c441faff1541676a57476" member_type: VOTER last_known_addr { host: "127.6.234.254" port: 44391 } }
I20260620 11:42:41.789165 10055 raft_consensus.cc:388] T 00000000000000000000000000000000 P 3c89e4953e5c441faff1541676a57476 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260620 11:42:41.789188 10055 raft_consensus.cc:482] T 00000000000000000000000000000000 P 3c89e4953e5c441faff1541676a57476 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260620 11:42:41.789256 10055 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 3c89e4953e5c441faff1541676a57476 [term 0 FOLLOWER]: Advancing to term 1
I20260620 11:42:41.789899 10055 raft_consensus.cc:504] T 00000000000000000000000000000000 P 3c89e4953e5c441faff1541676a57476 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3c89e4953e5c441faff1541676a57476" member_type: VOTER last_known_addr { host: "127.6.234.254" port: 44391 } }
I20260620 11:42:41.790016 10055 leader_election.cc:302] T 00000000000000000000000000000000 P 3c89e4953e5c441faff1541676a57476 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 3c89e4953e5c441faff1541676a57476; no voters:
I20260620 11:42:41.790167 10055 leader_election.cc:288] T 00000000000000000000000000000000 P 3c89e4953e5c441faff1541676a57476 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260620 11:42:41.790238 10060 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 3c89e4953e5c441faff1541676a57476 [term 1 FOLLOWER]: Leader election won for term 1
I20260620 11:42:41.790398 10060 raft_consensus.cc:686] T 00000000000000000000000000000000 P 3c89e4953e5c441faff1541676a57476 [term 1 LEADER]: Becoming Leader. State: Replica: 3c89e4953e5c441faff1541676a57476, State: Running, Role: LEADER
I20260620 11:42:41.790409 10055 sys_catalog.cc:564] T 00000000000000000000000000000000 P 3c89e4953e5c441faff1541676a57476 [sys.catalog]: configured and running, proceeding with master startup.
I20260620 11:42:41.790555 10060 consensus_queue.cc:237] T 00000000000000000000000000000000 P 3c89e4953e5c441faff1541676a57476 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3c89e4953e5c441faff1541676a57476" member_type: VOTER last_known_addr { host: "127.6.234.254" port: 44391 } }
I20260620 11:42:41.790849 10060 sys_catalog.cc:455] T 00000000000000000000000000000000 P 3c89e4953e5c441faff1541676a57476 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 3c89e4953e5c441faff1541676a57476. Latest consensus state: current_term: 1 leader_uuid: "3c89e4953e5c441faff1541676a57476" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3c89e4953e5c441faff1541676a57476" member_type: VOTER last_known_addr { host: "127.6.234.254" port: 44391 } } }
I20260620 11:42:41.790858 10062 sys_catalog.cc:455] T 00000000000000000000000000000000 P 3c89e4953e5c441faff1541676a57476 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "3c89e4953e5c441faff1541676a57476" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3c89e4953e5c441faff1541676a57476" member_type: VOTER last_known_addr { host: "127.6.234.254" port: 44391 } } }
I20260620 11:42:41.790954 10062 sys_catalog.cc:458] T 00000000000000000000000000000000 P 3c89e4953e5c441faff1541676a57476 [sys.catalog]: This master's current role is: LEADER
I20260620 11:42:41.790947 10060 sys_catalog.cc:458] T 00000000000000000000000000000000 P 3c89e4953e5c441faff1541676a57476 [sys.catalog]: This master's current role is: LEADER
I20260620 11:42:41.791185 10066 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260620 11:42:41.791677 10066 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260620 11:42:41.793483 10066 catalog_manager.cc:1347] Generated new cluster ID: 9096c100a5b845619f3a9706d038a720
I20260620 11:42:41.793535 10066 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
Jun 20 11:42:41 dist-test-slave-cjc6 krb5kdc[9974](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781955761, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.6.234.254@KRBTEST.COM
I20260620 11:42:41.799548 10059 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 11:42:41.786990 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59508 (local address 127.6.234.254:44391)
0620 11:42:41.787186 (+ 196us) server_negotiation.cc:206] Beginning negotiation
0620 11:42:41.787196 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0620 11:42:41.787229 (+ 33us) server_negotiation.cc:407] Connection header received
0620 11:42:41.788618 (+ 1389us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 11:42:41.788627 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 11:42:41.788690 (+ 63us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 11:42:41.788853 (+ 163us) server_negotiation.cc:226] Negotiated authn=SASL
0620 11:42:41.789641 (+ 788us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 11:42:41.791085 (+ 1444us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 11:42:41.792261 (+ 1176us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 11:42:41.792824 (+ 563us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 11:42:41.792950 (+ 126us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 11:42:41.795961 (+ 3011us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 11:42:41.795965 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 11:42:41.795965 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 11:42:41.795976 (+ 11us) server_negotiation.cc:922] calling sasl_server_start()
0620 11:42:41.797872 (+ 1896us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 11:42:41.798346 (+ 474us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 11:42:41.798348 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 11:42:41.798349 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 11:42:41.798468 (+ 119us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 11:42:41.798681 (+ 213us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 11:42:41.798682 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 11:42:41.798682 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 11:42:41.798974 (+ 292us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 11:42:41.799118 (+ 144us) server_negotiation.cc:1035] Waiting for connection context
0620 11:42:41.799242 (+ 124us) server_negotiation.cc:299] Negotiation successful
0620 11:42:41.799367 (+ 125us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":143,"thread_start_us":83,"threads_started":1}
I20260620 11:42:41.800601 7083 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260620 11:42:41.804642 10066 catalog_manager.cc:1370] Generated new certificate authority record
I20260620 11:42:41.805370 10066 catalog_manager.cc:1504] Loading token signing keys...
I20260620 11:42:41.812623 10066 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 3c89e4953e5c441faff1541676a57476: Generated new TSK 0
I20260620 11:42:41.812791 10066 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260620 11:42:41.824570 10059 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 11:42:41.820673 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59522 (local address 127.6.234.254:44391)
0620 11:42:41.820736 (+ 63us) server_negotiation.cc:206] Beginning negotiation
0620 11:42:41.820738 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 11:42:41.820875 (+ 137us) server_negotiation.cc:407] Connection header received
0620 11:42:41.820957 (+ 82us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 11:42:41.820958 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 11:42:41.820969 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 11:42:41.821027 (+ 58us) server_negotiation.cc:226] Negotiated authn=SASL
0620 11:42:41.821369 (+ 342us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 11:42:41.822210 (+ 841us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 11:42:41.823003 (+ 793us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 11:42:41.823281 (+ 278us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 11:42:41.823316 (+ 35us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 11:42:41.824060 (+ 744us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 11:42:41.824061 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 11:42:41.824062 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 11:42:41.824322 (+ 260us) server_negotiation.cc:922] calling sasl_server_start()
0620 11:42:41.824341 (+ 19us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 11:42:41.824348 (+ 7us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 11:42:41.824414 (+ 66us) server_negotiation.cc:1035] Waiting for connection context
0620 11:42:41.824461 (+ 47us) server_negotiation.cc:299] Negotiation successful
0620 11:42:41.824489 (+ 28us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":39}
I20260620 11:42:41.826982 7083 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task1JRg7N/build/release/bin/kudu with pid 9987
2026-06-20T11:42:41Z chronyd exiting
[ OK ] SecurityITest.SaslPlainFallback (1314 ms)
[ RUN ] SecurityITest.TestUnauthorizedClientKerberosCredentials
Loading random data
Initializing database '/tmp/dist-test-task1JRg7N/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781955746769051-7083-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 20 11:42:41 dist-test-slave-cjc6 krb5kdc[10088](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 20 11:42:41 dist-test-slave-cjc6 krb5kdc[10088](info): set up 2 sockets
Jun 20 11:42:41 dist-test-slave-cjc6 krb5kdc[10088](info): commencing operation
krb5kdc: starting...
W20260620 11:42:42.857435 7083 mini_kdc.cc:121] Time spent starting KDC: real 1.013s user 0.000s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task1JRg7N/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781955746769051-7083-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task1JRg7N/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781955746769051-7083-0/krb5kdc/test-user.keytab.
Jun 20 11:42:42 dist-test-slave-cjc6 krb5kdc[10088](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781955762, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-20T11:42:42Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-20T11:42:42Z Disabled control of system clock
WARNING: no policy specified for kudu/127.6.234.254@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.6.234.254@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.6.234.254 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task1JRg7N/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781955746769051-7083-0/krb5kdc/kudu_127.6.234.254.keytab.
Entry for principal kudu/127.6.234.254 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task1JRg7N/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781955746769051-7083-0/krb5kdc/kudu_127.6.234.254.keytab.
I20260620 11:42:42.968314 7083 external_mini_cluster.cc:1326] Running /tmp/dist-test-task1JRg7N/build/release/bin/kudu
/tmp/dist-test-task1JRg7N/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task1JRg7N/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781955746769051-7083-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task1JRg7N/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781955746769051-7083-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task1JRg7N/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781955746769051-7083-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task1JRg7N/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781955746769051-7083-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.6.234.254:40729
--webserver_interface=127.6.234.254
--webserver_port=0
--keytab_file=/tmp/dist-test-task1JRg7N/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781955746769051-7083-0/krb5kdc/kudu_127.6.234.254.keytab
--principal=kudu/127.6.234.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.6.234.212:39363
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.6.234.254:40729
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task1JRg7N/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781955746769051-7083-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task1JRg7N/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781955746769051-7083-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task1JRg7N/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781955746769051-7083-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 11:42:43.047510 10101 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 11:42:43.047753 10101 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 11:42:43.047791 10101 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 11:42:43.048923 10101 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 11:42:43.048997 10101 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 11:42:43.049021 10101 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 11:42:43.049044 10101 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 11:42:43.049072 10101 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 11:42:43.050526 10101 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.6.234.212:39363
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task1JRg7N/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781955746769051-7083-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task1JRg7N/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781955746769051-7083-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.6.234.254:40729
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task1JRg7N/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781955746769051-7083-0/krb5kdc/kudu_127.6.234.254.keytab
--principal=kudu/127.6.234.254
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.6.234.254:40729
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task1JRg7N/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781955746769051-7083-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.6.234.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.10101
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task1JRg7N/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781955746769051-7083-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 20 Jun 2026 10:32:37 UTC on ccycloud.quasar-bgyofv.root.comops.site
build id 4457173
I20260620 11:42:43.050791 10101 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 11:42:43.051009 10101 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 11:42:43.056095 10109 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 11:42:43.056082 10106 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 11:42:43.056082 10107 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 11:42:43.056106 10101 server_base.cc:1034] running on GCE node
I20260620 11:42:43.056559 10101 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 11:42:43.056874 10101 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 11:42:43.058040 10101 hybrid_clock.cc:648] HybridClock initialized: now 1781955763058026 us; error 44 us; skew 500 ppm
Jun 20 11:42:43 dist-test-slave-cjc6 krb5kdc[10088](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781955763, etypes {rep=17 tkt=17 ses=17}, kudu/127.6.234.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 11:42:43.065335 10101 init.cc:377] Logged in from keytab as kudu/127.6.234.254@KRBTEST.COM (short username kudu)
I20260620 11:42:43.065929 10101 webserver.cc:460] Webserver started at http://127.6.234.254:40225/ using document root <none> and password file <none>
I20260620 11:42:43.066195 10101 fs_manager.cc:359] Metadata directory not provided
I20260620 11:42:43.066267 10101 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 11:42:43.066418 10101 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 11:42:43.067497 10101 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task1JRg7N/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781955746769051-7083-0/minicluster-data/master-0/data/instance:
uuid: "c77e1456789d4299b7abc34633b35482"
format_stamp: "Formatted at 2026-06-20 11:42:43 on dist-test-slave-cjc6"
server_key: "db6a421da596d0768b96624da3837f04"
server_key_iv: "49d5232a23afbb6ed116dac592b438d0"
server_key_version: "encryptionkey@0"
I20260620 11:42:43.067870 10101 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task1JRg7N/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781955746769051-7083-0/minicluster-data/master-0/wal/instance:
uuid: "c77e1456789d4299b7abc34633b35482"
format_stamp: "Formatted at 2026-06-20 11:42:43 on dist-test-slave-cjc6"
server_key: "db6a421da596d0768b96624da3837f04"
server_key_iv: "49d5232a23afbb6ed116dac592b438d0"
server_key_version: "encryptionkey@0"
I20260620 11:42:43.069242 10101 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260620 11:42:43.070019 10116 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 11:42:43.070317 10101 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260620 11:42:43.070427 10101 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task1JRg7N/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781955746769051-7083-0/minicluster-data/master-0/data,/tmp/dist-test-task1JRg7N/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781955746769051-7083-0/minicluster-data/master-0/wal
uuid: "c77e1456789d4299b7abc34633b35482"
format_stamp: "Formatted at 2026-06-20 11:42:43 on dist-test-slave-cjc6"
server_key: "db6a421da596d0768b96624da3837f04"
server_key_iv: "49d5232a23afbb6ed116dac592b438d0"
server_key_version: "encryptionkey@0"
I20260620 11:42:43.070503 10101 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task1JRg7N/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781955746769051-7083-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task1JRg7N/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781955746769051-7083-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task1JRg7N/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781955746769051-7083-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 11:42:43.078740 10101 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 11:42:43.080494 10101 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 11:42:43.080648 10101 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 11:42:43.084621 10101 rpc_server.cc:307] RPC server started. Bound to: 127.6.234.254:40729
I20260620 11:42:43.084651 10168 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.6.234.254:40729 every 8 connection(s)
I20260620 11:42:43.085011 10101 server_base.cc:1166] Dumped server information to /tmp/dist-test-task1JRg7N/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781955746769051-7083-0/minicluster-data/master-0/data/info.pb
I20260620 11:42:43.085999 10169 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 11:42:43.088099 10169 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P c77e1456789d4299b7abc34633b35482: Bootstrap starting.
I20260620 11:42:43.088598 10169 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P c77e1456789d4299b7abc34633b35482: Neither blocks nor log segments found. Creating new log.
I20260620 11:42:43.088819 10169 log.cc:824] T 00000000000000000000000000000000 P c77e1456789d4299b7abc34633b35482: Log is configured to *not* fsync() on all Append() calls
I20260620 11:42:43.089435 10169 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P c77e1456789d4299b7abc34633b35482: No bootstrap required, opened a new log
I20260620 11:42:43.091017 10169 raft_consensus.cc:348] T 00000000000000000000000000000000 P c77e1456789d4299b7abc34633b35482 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c77e1456789d4299b7abc34633b35482" member_type: VOTER last_known_addr { host: "127.6.234.254" port: 40729 } }
I20260620 11:42:43.091148 10169 raft_consensus.cc:374] T 00000000000000000000000000000000 P c77e1456789d4299b7abc34633b35482 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 11:42:43.091171 10169 raft_consensus.cc:729] T 00000000000000000000000000000000 P c77e1456789d4299b7abc34633b35482 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c77e1456789d4299b7abc34633b35482, State: Initialized, Role: FOLLOWER
I20260620 11:42:43.091266 10169 consensus_queue.cc:260] T 00000000000000000000000000000000 P c77e1456789d4299b7abc34633b35482 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c77e1456789d4299b7abc34633b35482" member_type: VOTER last_known_addr { host: "127.6.234.254" port: 40729 } }
I20260620 11:42:43.091317 10169 raft_consensus.cc:388] T 00000000000000000000000000000000 P c77e1456789d4299b7abc34633b35482 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260620 11:42:43.091337 10169 raft_consensus.cc:482] T 00000000000000000000000000000000 P c77e1456789d4299b7abc34633b35482 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260620 11:42:43.091403 10169 raft_consensus.cc:3037] T 00000000000000000000000000000000 P c77e1456789d4299b7abc34633b35482 [term 0 FOLLOWER]: Advancing to term 1
I20260620 11:42:43.092145 10169 raft_consensus.cc:504] T 00000000000000000000000000000000 P c77e1456789d4299b7abc34633b35482 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c77e1456789d4299b7abc34633b35482" member_type: VOTER last_known_addr { host: "127.6.234.254" port: 40729 } }
I20260620 11:42:43.092255 10169 leader_election.cc:302] T 00000000000000000000000000000000 P c77e1456789d4299b7abc34633b35482 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: c77e1456789d4299b7abc34633b35482; no voters:
I20260620 11:42:43.092427 10169 leader_election.cc:288] T 00000000000000000000000000000000 P c77e1456789d4299b7abc34633b35482 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260620 11:42:43.092494 10172 raft_consensus.cc:2781] T 00000000000000000000000000000000 P c77e1456789d4299b7abc34633b35482 [term 1 FOLLOWER]: Leader election won for term 1
I20260620 11:42:43.092697 10169 sys_catalog.cc:564] T 00000000000000000000000000000000 P c77e1456789d4299b7abc34633b35482 [sys.catalog]: configured and running, proceeding with master startup.
I20260620 11:42:43.092732 10172 raft_consensus.cc:686] T 00000000000000000000000000000000 P c77e1456789d4299b7abc34633b35482 [term 1 LEADER]: Becoming Leader. State: Replica: c77e1456789d4299b7abc34633b35482, State: Running, Role: LEADER
I20260620 11:42:43.092828 7083 external_mini_cluster.cc:1388] Started /tmp/dist-test-task1JRg7N/build/release/bin/kudu as pid 10101
I20260620 11:42:43.092854 10172 consensus_queue.cc:237] T 00000000000000000000000000000000 P c77e1456789d4299b7abc34633b35482 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c77e1456789d4299b7abc34633b35482" member_type: VOTER last_known_addr { host: "127.6.234.254" port: 40729 } }
I20260620 11:42:43.092921 7083 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task1JRg7N/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781955746769051-7083-0/minicluster-data/master-0/wal/instance
I20260620 11:42:43.093096 7083 external_mini_cluster.cc:1428] Setting key f14068378fbcfa5ca1bc486789a9552e
I20260620 11:42:43.093241 10173 sys_catalog.cc:455] T 00000000000000000000000000000000 P c77e1456789d4299b7abc34633b35482 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "c77e1456789d4299b7abc34633b35482" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c77e1456789d4299b7abc34633b35482" member_type: VOTER last_known_addr { host: "127.6.234.254" port: 40729 } } }
I20260620 11:42:43.093256 10174 sys_catalog.cc:455] T 00000000000000000000000000000000 P c77e1456789d4299b7abc34633b35482 [sys.catalog]: SysCatalogTable state changed. Reason: New leader c77e1456789d4299b7abc34633b35482. Latest consensus state: current_term: 1 leader_uuid: "c77e1456789d4299b7abc34633b35482" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c77e1456789d4299b7abc34633b35482" member_type: VOTER last_known_addr { host: "127.6.234.254" port: 40729 } } }
I20260620 11:42:43.093348 10173 sys_catalog.cc:458] T 00000000000000000000000000000000 P c77e1456789d4299b7abc34633b35482 [sys.catalog]: This master's current role is: LEADER
I20260620 11:42:43.093356 10174 sys_catalog.cc:458] T 00000000000000000000000000000000 P c77e1456789d4299b7abc34633b35482 [sys.catalog]: This master's current role is: LEADER
I20260620 11:42:43.093600 10181 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260620 11:42:43.094187 10181 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260620 11:42:43.095809 10181 catalog_manager.cc:1347] Generated new cluster ID: 2a9bab0ba4d442a385ab361439a68cc1
I20260620 11:42:43.095870 10181 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
Jun 20 11:42:43 dist-test-slave-cjc6 krb5kdc[10088](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781955762, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.6.234.254@KRBTEST.COM
I20260620 11:42:43.110949 10185 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 11:42:43.093721 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45522 (local address 127.6.234.254:40729)
0620 11:42:43.094143 (+ 422us) server_negotiation.cc:206] Beginning negotiation
0620 11:42:43.094152 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0620 11:42:43.094184 (+ 32us) server_negotiation.cc:407] Connection header received
0620 11:42:43.094311 (+ 127us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 11:42:43.094319 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 11:42:43.094375 (+ 56us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 11:42:43.094502 (+ 127us) server_negotiation.cc:226] Negotiated authn=SASL
0620 11:42:43.095275 (+ 773us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 11:42:43.096597 (+ 1322us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 11:42:43.097641 (+ 1044us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 11:42:43.098291 (+ 650us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 11:42:43.098419 (+ 128us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 11:42:43.105311 (+ 6892us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 11:42:43.105316 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 11:42:43.105317 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 11:42:43.105329 (+ 12us) server_negotiation.cc:922] calling sasl_server_start()
0620 11:42:43.108730 (+ 3401us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 11:42:43.109140 (+ 410us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 11:42:43.109144 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 11:42:43.109144 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 11:42:43.109249 (+ 105us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 11:42:43.109451 (+ 202us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 11:42:43.109451 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 11:42:43.109452 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 11:42:43.109949 (+ 497us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 11:42:43.110114 (+ 165us) server_negotiation.cc:1035] Waiting for connection context
0620 11:42:43.110559 (+ 445us) server_negotiation.cc:299] Negotiation successful
0620 11:42:43.110703 (+ 144us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":361,"thread_start_us":77,"threads_started":1}
I20260620 11:42:43.123566 10181 catalog_manager.cc:1370] Generated new certificate authority record
I20260620 11:42:43.124298 10181 catalog_manager.cc:1504] Loading token signing keys...
I20260620 11:42:43.131891 10181 catalog_manager.cc:5868] T 00000000000000000000000000000000 P c77e1456789d4299b7abc34633b35482: Generated new TSK 0
I20260620 11:42:43.132077 10181 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.6.234.193@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.6.234.193@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.6.234.193 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task1JRg7N/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781955746769051-7083-0/krb5kdc/kudu_127.6.234.193.keytab.
Entry for principal kudu/127.6.234.193 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task1JRg7N/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781955746769051-7083-0/krb5kdc/kudu_127.6.234.193.keytab.
I20260620 11:42:43.158109 7083 external_mini_cluster.cc:1326] Running /tmp/dist-test-task1JRg7N/build/release/bin/kudu
/tmp/dist-test-task1JRg7N/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task1JRg7N/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781955746769051-7083-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-task1JRg7N/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781955746769051-7083-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task1JRg7N/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781955746769051-7083-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-task1JRg7N/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781955746769051-7083-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.6.234.193:0
--local_ip_for_outbound_sockets=127.6.234.193
--webserver_interface=127.6.234.193
--webserver_port=0
--tserver_master_addrs=127.6.234.254:40729
--keytab_file=/tmp/dist-test-task1JRg7N/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781955746769051-7083-0/krb5kdc/kudu_127.6.234.193.keytab
--principal=kudu/127.6.234.193
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.6.234.212:39363
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task1JRg7N/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781955746769051-7083-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task1JRg7N/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781955746769051-7083-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task1JRg7N/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781955746769051-7083-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 11:42:43.237670 10195 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 11:42:43.237902 10195 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 11:42:43.237926 10195 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 11:42:43.239081 10195 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 11:42:43.239122 10195 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 11:42:43.239148 10195 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.6.234.193
I20260620 11:42:43.240576 10195 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.6.234.212:39363
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task1JRg7N/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781955746769051-7083-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-task1JRg7N/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781955746769051-7083-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task1JRg7N/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781955746769051-7083-0/krb5kdc/kudu_127.6.234.193.keytab
--principal=kudu/127.6.234.193
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.6.234.193:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task1JRg7N/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781955746769051-7083-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.6.234.193
--webserver_port=0
--tserver_master_addrs=127.6.234.254:40729
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.10195
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.6.234.193
--log_dir=/tmp/dist-test-task1JRg7N/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781955746769051-7083-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 20 Jun 2026 10:32:37 UTC on ccycloud.quasar-bgyofv.root.comops.site
build id 4457173
I20260620 11:42:43.240825 10195 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 11:42:43.241050 10195 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 11:42:43.245904 10203 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 11:42:43.246006 10195 server_base.cc:1034] running on GCE node
W20260620 11:42:43.245920 10200 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 11:42:43.245920 10201 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 11:42:43.246301 10195 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 11:42:43.246548 10195 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 11:42:43.247707 10195 hybrid_clock.cc:648] HybridClock initialized: now 1781955763247685 us; error 45 us; skew 500 ppm
Jun 20 11:42:43 dist-test-slave-cjc6 krb5kdc[10088](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781955763, etypes {rep=17 tkt=17 ses=17}, kudu/127.6.234.193@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 11:42:43.250080 10195 init.cc:377] Logged in from keytab as kudu/127.6.234.193@KRBTEST.COM (short username kudu)
I20260620 11:42:43.250569 10195 webserver.cc:460] Webserver started at http://127.6.234.193:42577/ using document root <none> and password file <none>
I20260620 11:42:43.250752 10195 fs_manager.cc:359] Metadata directory not provided
I20260620 11:42:43.250810 10195 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 11:42:43.250897 10195 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 11:42:43.251868 10195 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task1JRg7N/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781955746769051-7083-0/minicluster-data/ts-0/data/instance:
uuid: "2449e7841eff457eb5a52491a90d97a6"
format_stamp: "Formatted at 2026-06-20 11:42:43 on dist-test-slave-cjc6"
server_key: "fe68228499969440c21025cf39055cc1"
server_key_iv: "dea974a1ab7535adb3e8b436ab3a03d6"
server_key_version: "encryptionkey@0"
I20260620 11:42:43.252262 10195 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task1JRg7N/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781955746769051-7083-0/minicluster-data/ts-0/wal/instance:
uuid: "2449e7841eff457eb5a52491a90d97a6"
format_stamp: "Formatted at 2026-06-20 11:42:43 on dist-test-slave-cjc6"
server_key: "fe68228499969440c21025cf39055cc1"
server_key_iv: "dea974a1ab7535adb3e8b436ab3a03d6"
server_key_version: "encryptionkey@0"
I20260620 11:42:43.253506 10195 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.001s
I20260620 11:42:43.254257 10210 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 11:42:43.254452 10195 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.000s
I20260620 11:42:43.254528 10195 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task1JRg7N/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781955746769051-7083-0/minicluster-data/ts-0/data,/tmp/dist-test-task1JRg7N/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781955746769051-7083-0/minicluster-data/ts-0/wal
uuid: "2449e7841eff457eb5a52491a90d97a6"
format_stamp: "Formatted at 2026-06-20 11:42:43 on dist-test-slave-cjc6"
server_key: "fe68228499969440c21025cf39055cc1"
server_key_iv: "dea974a1ab7535adb3e8b436ab3a03d6"
server_key_version: "encryptionkey@0"
I20260620 11:42:43.254580 10195 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task1JRg7N/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781955746769051-7083-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-task1JRg7N/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781955746769051-7083-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-task1JRg7N/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781955746769051-7083-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 11:42:43.271584 10195 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 11:42:43.273888 10195 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 11:42:43.274044 10195 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 11:42:43.274297 10195 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 11:42:43.274614 10195 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 11:42:43.274654 10195 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 11:42:43.274678 10195 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 11:42:43.274721 10195 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 11:42:43.280289 10195 rpc_server.cc:307] RPC server started. Bound to: 127.6.234.193:40733
I20260620 11:42:43.280364 10323 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.6.234.193:40733 every 8 connection(s)
I20260620 11:42:43.280684 10195 server_base.cc:1166] Dumped server information to /tmp/dist-test-task1JRg7N/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781955746769051-7083-0/minicluster-data/ts-0/data/info.pb
I20260620 11:42:43.282430 7083 external_mini_cluster.cc:1388] Started /tmp/dist-test-task1JRg7N/build/release/bin/kudu as pid 10195
I20260620 11:42:43.282521 7083 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task1JRg7N/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781955746769051-7083-0/minicluster-data/ts-0/wal/instance
I20260620 11:42:43.282718 7083 external_mini_cluster.cc:1428] Setting key d44208aeb3bcbe6ae83a0fe5132f76eb
Jun 20 11:42:43 dist-test-slave-cjc6 krb5kdc[10088](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781955763, etypes {rep=17 tkt=17 ses=17}, kudu/127.6.234.193@KRBTEST.COM for kudu/127.6.234.254@KRBTEST.COM
I20260620 11:42:43.290537 10185 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 11:42:43.281492 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.6.234.193:45803 (local address 127.6.234.254:40729)
0620 11:42:43.281549 (+ 57us) server_negotiation.cc:206] Beginning negotiation
0620 11:42:43.281551 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 11:42:43.281664 (+ 113us) server_negotiation.cc:407] Connection header received
0620 11:42:43.282268 (+ 604us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 11:42:43.282270 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 11:42:43.282284 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 11:42:43.282364 (+ 80us) server_negotiation.cc:226] Negotiated authn=SASL
0620 11:42:43.283427 (+ 1063us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 11:42:43.283920 (+ 493us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 11:42:43.284913 (+ 993us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 11:42:43.285118 (+ 205us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 11:42:43.285206 (+ 88us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 11:42:43.287557 (+ 2351us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 11:42:43.287559 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 11:42:43.287559 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 11:42:43.287565 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0620 11:42:43.288995 (+ 1430us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 11:42:43.289495 (+ 500us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 11:42:43.289496 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 11:42:43.289496 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 11:42:43.289550 (+ 54us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 11:42:43.289866 (+ 316us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 11:42:43.289867 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 11:42:43.289868 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 11:42:43.289987 (+ 119us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 11:42:43.290049 (+ 62us) server_negotiation.cc:1035] Waiting for connection context
0620 11:42:43.290400 (+ 351us) server_negotiation.cc:299] Negotiation successful
0620 11:42:43.290474 (+ 74us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":23}
I20260620 11:42:43.290776 10326 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 11:42:43.281378 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.234.254:40729 (local address 127.6.234.193:45803)
0620 11:42:43.281547 (+ 169us) negotiation.cc:107] Waiting for socket to connect
0620 11:42:43.281588 (+ 41us) client_negotiation.cc:174] Beginning negotiation
0620 11:42:43.282140 (+ 552us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 11:42:43.282434 (+ 294us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 11:42:43.282438 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 11:42:43.283124 (+ 686us) client_negotiation.cc:189] Negotiated authn=SASL
0620 11:42:43.283314 (+ 190us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 11:42:43.283319 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 11:42:43.284017 (+ 698us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 11:42:43.284019 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 11:42:43.284820 (+ 801us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 11:42:43.284822 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 11:42:43.285214 (+ 392us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 11:42:43.285215 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 11:42:43.285345 (+ 130us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 11:42:43.285900 (+ 555us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 11:42:43.285908 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0620 11:42:43.287447 (+ 1539us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 11:42:43.289083 (+ 1636us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 11:42:43.289091 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 11:42:43.289093 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0620 11:42:43.289412 (+ 319us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 11:42:43.289623 (+ 211us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 11:42:43.289624 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 11:42:43.289625 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0620 11:42:43.289793 (+ 168us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 11:42:43.290065 (+ 272us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 11:42:43.290072 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 11:42:43.290242 (+ 170us) client_negotiation.cc:769] Sending connection context
0620 11:42:43.290365 (+ 123us) client_negotiation.cc:240] Negotiation successful
0620 11:42:43.290552 (+ 187us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":128,"thread_start_us":99,"threads_started":1}
WARNING: no policy specified for kudu/127.6.234.194@KRBTEST.COM; defaulting to no policy
I20260620 11:42:43.291327 10324 heartbeater.cc:347] Connected to a master server at 127.6.234.254:40729
I20260620 11:42:43.291411 10324 heartbeater.cc:464] Registering TS with master...
I20260620 11:42:43.291548 10324 heartbeater.cc:511] Master 127.6.234.254:40729 requested a full tablet report, sending...
I20260620 11:42:43.292095 10133 ts_manager.cc:194] Registered new tserver with Master: 2449e7841eff457eb5a52491a90d97a6 (127.6.234.193:40733)
I20260620 11:42:43.292768 10133 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.6.234.193@KRBTEST.COM'} at 127.6.234.193:45803
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.6.234.194@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.6.234.194 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task1JRg7N/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781955746769051-7083-0/krb5kdc/kudu_127.6.234.194.keytab.
Entry for principal kudu/127.6.234.194 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task1JRg7N/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781955746769051-7083-0/krb5kdc/kudu_127.6.234.194.keytab.
I20260620 11:42:43.306465 7083 external_mini_cluster.cc:1326] Running /tmp/dist-test-task1JRg7N/build/release/bin/kudu
/tmp/dist-test-task1JRg7N/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task1JRg7N/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781955746769051-7083-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-task1JRg7N/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781955746769051-7083-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task1JRg7N/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781955746769051-7083-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-task1JRg7N/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781955746769051-7083-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.6.234.194:0
--local_ip_for_outbound_sockets=127.6.234.194
--webserver_interface=127.6.234.194
--webserver_port=0
--tserver_master_addrs=127.6.234.254:40729
--keytab_file=/tmp/dist-test-task1JRg7N/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781955746769051-7083-0/krb5kdc/kudu_127.6.234.194.keytab
--principal=kudu/127.6.234.194
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.6.234.212:39363
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task1JRg7N/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781955746769051-7083-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task1JRg7N/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781955746769051-7083-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task1JRg7N/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781955746769051-7083-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 11:42:43.385167 10329 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 11:42:43.385360 10329 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 11:42:43.385382 10329 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 11:42:43.386597 10329 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 11:42:43.386641 10329 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 11:42:43.386667 10329 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.6.234.194
I20260620 11:42:43.388064 10329 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.6.234.212:39363
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task1JRg7N/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781955746769051-7083-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-task1JRg7N/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781955746769051-7083-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task1JRg7N/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781955746769051-7083-0/krb5kdc/kudu_127.6.234.194.keytab
--principal=kudu/127.6.234.194
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.6.234.194:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task1JRg7N/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781955746769051-7083-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.6.234.194
--webserver_port=0
--tserver_master_addrs=127.6.234.254:40729
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.10329
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.6.234.194
--log_dir=/tmp/dist-test-task1JRg7N/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781955746769051-7083-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 20 Jun 2026 10:32:37 UTC on ccycloud.quasar-bgyofv.root.comops.site
build id 4457173
I20260620 11:42:43.388293 10329 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 11:42:43.388536 10329 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 11:42:43.393323 10334 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 11:42:43.393324 10337 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 11:42:43.393324 10335 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 11:42:43.393611 10329 server_base.cc:1034] running on GCE node
I20260620 11:42:43.393837 10329 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 11:42:43.394068 10329 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 11:42:43.395234 10329 hybrid_clock.cc:648] HybridClock initialized: now 1781955763395226 us; error 38 us; skew 500 ppm
Jun 20 11:42:43 dist-test-slave-cjc6 krb5kdc[10088](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781955763, etypes {rep=17 tkt=17 ses=17}, kudu/127.6.234.194@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 11:42:43.397591 10329 init.cc:377] Logged in from keytab as kudu/127.6.234.194@KRBTEST.COM (short username kudu)
I20260620 11:42:43.398044 10329 webserver.cc:460] Webserver started at http://127.6.234.194:41911/ using document root <none> and password file <none>
I20260620 11:42:43.398231 10329 fs_manager.cc:359] Metadata directory not provided
I20260620 11:42:43.398270 10329 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 11:42:43.398355 10329 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 11:42:43.399291 10329 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task1JRg7N/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781955746769051-7083-0/minicluster-data/ts-1/data/instance:
uuid: "547837b10974492abb9173cd7bb457a7"
format_stamp: "Formatted at 2026-06-20 11:42:43 on dist-test-slave-cjc6"
server_key: "fc099f4e480948282755d7940a7a8eb5"
server_key_iv: "ff9b70f676cc560882e29fca305d10dd"
server_key_version: "encryptionkey@0"
I20260620 11:42:43.399569 10329 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task1JRg7N/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781955746769051-7083-0/minicluster-data/ts-1/wal/instance:
uuid: "547837b10974492abb9173cd7bb457a7"
format_stamp: "Formatted at 2026-06-20 11:42:43 on dist-test-slave-cjc6"
server_key: "fc099f4e480948282755d7940a7a8eb5"
server_key_iv: "ff9b70f676cc560882e29fca305d10dd"
server_key_version: "encryptionkey@0"
I20260620 11:42:43.400763 10329 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260620 11:42:43.401407 10344 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 11:42:43.401583 10329 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260620 11:42:43.401666 10329 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task1JRg7N/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781955746769051-7083-0/minicluster-data/ts-1/data,/tmp/dist-test-task1JRg7N/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781955746769051-7083-0/minicluster-data/ts-1/wal
uuid: "547837b10974492abb9173cd7bb457a7"
format_stamp: "Formatted at 2026-06-20 11:42:43 on dist-test-slave-cjc6"
server_key: "fc099f4e480948282755d7940a7a8eb5"
server_key_iv: "ff9b70f676cc560882e29fca305d10dd"
server_key_version: "encryptionkey@0"
I20260620 11:42:43.401769 10329 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task1JRg7N/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781955746769051-7083-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-task1JRg7N/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781955746769051-7083-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-task1JRg7N/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781955746769051-7083-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 11:42:43.441833 10329 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 11:42:43.444186 10329 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 11:42:43.444298 10329 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 11:42:43.444482 10329 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 11:42:43.444813 10329 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 11:42:43.444849 10329 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 11:42:43.445036 10329 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 11:42:43.445081 10329 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 11:42:43.450764 10329 rpc_server.cc:307] RPC server started. Bound to: 127.6.234.194:36061
I20260620 11:42:43.450877 10457 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.6.234.194:36061 every 8 connection(s)
I20260620 11:42:43.451179 10329 server_base.cc:1166] Dumped server information to /tmp/dist-test-task1JRg7N/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781955746769051-7083-0/minicluster-data/ts-1/data/info.pb
Jun 20 11:42:43 dist-test-slave-cjc6 krb5kdc[10088](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781955763, etypes {rep=17 tkt=17 ses=17}, kudu/127.6.234.194@KRBTEST.COM for kudu/127.6.234.254@KRBTEST.COM
I20260620 11:42:43.460098 10185 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 11:42:43.451954 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.6.234.194:46667 (local address 127.6.234.254:40729)
0620 11:42:43.452018 (+ 64us) server_negotiation.cc:206] Beginning negotiation
0620 11:42:43.452020 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 11:42:43.452195 (+ 175us) server_negotiation.cc:407] Connection header received
0620 11:42:43.452673 (+ 478us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 11:42:43.452674 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 11:42:43.452684 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 11:42:43.452738 (+ 54us) server_negotiation.cc:226] Negotiated authn=SASL
0620 11:42:43.453640 (+ 902us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 11:42:43.454187 (+ 547us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 11:42:43.455158 (+ 971us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 11:42:43.455345 (+ 187us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 11:42:43.455396 (+ 51us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 11:42:43.457476 (+ 2080us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 11:42:43.457477 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 11:42:43.457478 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 11:42:43.457484 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0620 11:42:43.458997 (+ 1513us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 11:42:43.459351 (+ 354us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 11:42:43.459352 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 11:42:43.459352 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 11:42:43.459400 (+ 48us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 11:42:43.459602 (+ 202us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 11:42:43.459603 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 11:42:43.459603 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 11:42:43.459700 (+ 97us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 11:42:43.459748 (+ 48us) server_negotiation.cc:1035] Waiting for connection context
0620 11:42:43.459972 (+ 224us) server_negotiation.cc:299] Negotiation successful
0620 11:42:43.460033 (+ 61us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":37}
I20260620 11:42:43.460279 10460 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 11:42:43.451860 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.234.254:40729 (local address 127.6.234.194:46667)
0620 11:42:43.452076 (+ 216us) negotiation.cc:107] Waiting for socket to connect
0620 11:42:43.452112 (+ 36us) client_negotiation.cc:174] Beginning negotiation
0620 11:42:43.452588 (+ 476us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 11:42:43.452817 (+ 229us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 11:42:43.452820 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 11:42:43.453376 (+ 556us) client_negotiation.cc:189] Negotiated authn=SASL
0620 11:42:43.453563 (+ 187us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 11:42:43.453568 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 11:42:43.454261 (+ 693us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 11:42:43.454262 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 11:42:43.455086 (+ 824us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 11:42:43.455088 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 11:42:43.455412 (+ 324us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 11:42:43.455413 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 11:42:43.455521 (+ 108us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 11:42:43.455965 (+ 444us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 11:42:43.455973 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0620 11:42:43.457396 (+ 1423us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 11:42:43.459070 (+ 1674us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 11:42:43.459076 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 11:42:43.459077 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0620 11:42:43.459284 (+ 207us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 11:42:43.459459 (+ 175us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 11:42:43.459460 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 11:42:43.459460 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0620 11:42:43.459549 (+ 89us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 11:42:43.459752 (+ 203us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 11:42:43.459756 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 11:42:43.459867 (+ 111us) client_negotiation.cc:769] Sending connection context
0620 11:42:43.459945 (+ 78us) client_negotiation.cc:240] Negotiation successful
0620 11:42:43.460057 (+ 112us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":183,"thread_start_us":102,"threads_started":1}
I20260620 11:42:43.460819 10458 heartbeater.cc:347] Connected to a master server at 127.6.234.254:40729
I20260620 11:42:43.460897 10458 heartbeater.cc:464] Registering TS with master...
I20260620 11:42:43.461032 10458 heartbeater.cc:511] Master 127.6.234.254:40729 requested a full tablet report, sending...
I20260620 11:42:43.461160 7083 external_mini_cluster.cc:1388] Started /tmp/dist-test-task1JRg7N/build/release/bin/kudu as pid 10329
I20260620 11:42:43.461278 7083 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task1JRg7N/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781955746769051-7083-0/minicluster-data/ts-1/wal/instance
I20260620 11:42:43.461443 10133 ts_manager.cc:194] Registered new tserver with Master: 547837b10974492abb9173cd7bb457a7 (127.6.234.194:36061)
I20260620 11:42:43.461472 7083 external_mini_cluster.cc:1428] Setting key d623b564622362020d7ffdbe2050a49f
I20260620 11:42:43.461992 10133 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.6.234.194@KRBTEST.COM'} at 127.6.234.194:46667
WARNING: no policy specified for kudu/127.6.234.195@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.6.234.195@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.6.234.195 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task1JRg7N/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781955746769051-7083-0/krb5kdc/kudu_127.6.234.195.keytab.
Entry for principal kudu/127.6.234.195 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task1JRg7N/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781955746769051-7083-0/krb5kdc/kudu_127.6.234.195.keytab.
I20260620 11:42:43.485294 7083 external_mini_cluster.cc:1326] Running /tmp/dist-test-task1JRg7N/build/release/bin/kudu
/tmp/dist-test-task1JRg7N/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task1JRg7N/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781955746769051-7083-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-task1JRg7N/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781955746769051-7083-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task1JRg7N/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781955746769051-7083-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-task1JRg7N/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781955746769051-7083-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.6.234.195:0
--local_ip_for_outbound_sockets=127.6.234.195
--webserver_interface=127.6.234.195
--webserver_port=0
--tserver_master_addrs=127.6.234.254:40729
--keytab_file=/tmp/dist-test-task1JRg7N/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781955746769051-7083-0/krb5kdc/kudu_127.6.234.195.keytab
--principal=kudu/127.6.234.195
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.6.234.212:39363
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task1JRg7N/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781955746769051-7083-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task1JRg7N/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781955746769051-7083-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task1JRg7N/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781955746769051-7083-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 11:42:43.564730 10463 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 11:42:43.564976 10463 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 11:42:43.565014 10463 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 11:42:43.566220 10463 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 11:42:43.566282 10463 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 11:42:43.566329 10463 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.6.234.195
I20260620 11:42:43.567742 10463 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.6.234.212:39363
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task1JRg7N/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781955746769051-7083-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-task1JRg7N/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781955746769051-7083-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task1JRg7N/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781955746769051-7083-0/krb5kdc/kudu_127.6.234.195.keytab
--principal=kudu/127.6.234.195
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.6.234.195:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task1JRg7N/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781955746769051-7083-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.6.234.195
--webserver_port=0
--tserver_master_addrs=127.6.234.254:40729
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.10463
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.6.234.195
--log_dir=/tmp/dist-test-task1JRg7N/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781955746769051-7083-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 20 Jun 2026 10:32:37 UTC on ccycloud.quasar-bgyofv.root.comops.site
build id 4457173
I20260620 11:42:43.568001 10463 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 11:42:43.568217 10463 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 11:42:43.573300 10469 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 11:42:43.573341 10468 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 11:42:43.573413 10463 server_base.cc:1034] running on GCE node
W20260620 11:42:43.573318 10471 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 11:42:43.573894 10463 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 11:42:43.574131 10463 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 11:42:43.575279 10463 hybrid_clock.cc:648] HybridClock initialized: now 1781955763575268 us; error 26 us; skew 500 ppm
Jun 20 11:42:43 dist-test-slave-cjc6 krb5kdc[10088](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781955763, etypes {rep=17 tkt=17 ses=17}, kudu/127.6.234.195@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 11:42:43.577907 10463 init.cc:377] Logged in from keytab as kudu/127.6.234.195@KRBTEST.COM (short username kudu)
I20260620 11:42:43.578382 10463 webserver.cc:460] Webserver started at http://127.6.234.195:35699/ using document root <none> and password file <none>
I20260620 11:42:43.578601 10463 fs_manager.cc:359] Metadata directory not provided
I20260620 11:42:43.578644 10463 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 11:42:43.578794 10463 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 11:42:43.579807 10463 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task1JRg7N/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781955746769051-7083-0/minicluster-data/ts-2/data/instance:
uuid: "cdac5698e97345f1b6ae236b67733273"
format_stamp: "Formatted at 2026-06-20 11:42:43 on dist-test-slave-cjc6"
server_key: "a00d78aa1d3b684ca467561a1fae66af"
server_key_iv: "e9cc6ef0f80e7b721f02901a6bbda598"
server_key_version: "encryptionkey@0"
I20260620 11:42:43.580155 10463 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task1JRg7N/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781955746769051-7083-0/minicluster-data/ts-2/wal/instance:
uuid: "cdac5698e97345f1b6ae236b67733273"
format_stamp: "Formatted at 2026-06-20 11:42:43 on dist-test-slave-cjc6"
server_key: "a00d78aa1d3b684ca467561a1fae66af"
server_key_iv: "e9cc6ef0f80e7b721f02901a6bbda598"
server_key_version: "encryptionkey@0"
I20260620 11:42:43.581456 10463 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260620 11:42:43.582258 10478 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 11:42:43.582465 10463 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.000s
I20260620 11:42:43.582537 10463 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task1JRg7N/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781955746769051-7083-0/minicluster-data/ts-2/data,/tmp/dist-test-task1JRg7N/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781955746769051-7083-0/minicluster-data/ts-2/wal
uuid: "cdac5698e97345f1b6ae236b67733273"
format_stamp: "Formatted at 2026-06-20 11:42:43 on dist-test-slave-cjc6"
server_key: "a00d78aa1d3b684ca467561a1fae66af"
server_key_iv: "e9cc6ef0f80e7b721f02901a6bbda598"
server_key_version: "encryptionkey@0"
I20260620 11:42:43.582626 10463 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task1JRg7N/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781955746769051-7083-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-task1JRg7N/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781955746769051-7083-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-task1JRg7N/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781955746769051-7083-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 11:42:43.595557 10463 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 11:42:43.597366 10463 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 11:42:43.597471 10463 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 11:42:43.597637 10463 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 11:42:43.597988 10463 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 11:42:43.598024 10463 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 11:42:43.598045 10463 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 11:42:43.598058 10463 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 11:42:43.603605 10463 rpc_server.cc:307] RPC server started. Bound to: 127.6.234.195:46335
I20260620 11:42:43.603654 10591 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.6.234.195:46335 every 8 connection(s)
I20260620 11:42:43.603987 10463 server_base.cc:1166] Dumped server information to /tmp/dist-test-task1JRg7N/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781955746769051-7083-0/minicluster-data/ts-2/data/info.pb
I20260620 11:42:43.609637 7083 external_mini_cluster.cc:1388] Started /tmp/dist-test-task1JRg7N/build/release/bin/kudu as pid 10463
I20260620 11:42:43.609761 7083 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task1JRg7N/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781955746769051-7083-0/minicluster-data/ts-2/wal/instance
I20260620 11:42:43.609975 7083 external_mini_cluster.cc:1428] Setting key 8a275280371142668e4d7c3035844c85
Jun 20 11:42:43 dist-test-slave-cjc6 krb5kdc[10088](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781955763, etypes {rep=17 tkt=17 ses=17}, kudu/127.6.234.195@KRBTEST.COM for kudu/127.6.234.254@KRBTEST.COM
I20260620 11:42:43.613612 10185 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 11:42:43.604722 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.6.234.195:51291 (local address 127.6.234.254:40729)
0620 11:42:43.604790 (+ 68us) server_negotiation.cc:206] Beginning negotiation
0620 11:42:43.604792 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 11:42:43.604999 (+ 207us) server_negotiation.cc:407] Connection header received
0620 11:42:43.605586 (+ 587us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 11:42:43.605588 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 11:42:43.605602 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 11:42:43.605652 (+ 50us) server_negotiation.cc:226] Negotiated authn=SASL
0620 11:42:43.606615 (+ 963us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 11:42:43.607108 (+ 493us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 11:42:43.608210 (+ 1102us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 11:42:43.608410 (+ 200us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 11:42:43.608496 (+ 86us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 11:42:43.610863 (+ 2367us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 11:42:43.610865 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 11:42:43.610865 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 11:42:43.610873 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0620 11:42:43.612256 (+ 1383us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 11:42:43.612664 (+ 408us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 11:42:43.612665 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 11:42:43.612666 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 11:42:43.612726 (+ 60us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 11:42:43.612987 (+ 261us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 11:42:43.612988 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 11:42:43.612989 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 11:42:43.613104 (+ 115us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 11:42:43.613178 (+ 74us) server_negotiation.cc:1035] Waiting for connection context
0620 11:42:43.613445 (+ 267us) server_negotiation.cc:299] Negotiation successful
0620 11:42:43.613521 (+ 76us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":39}
I20260620 11:42:43.613835 10594 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 11:42:43.604705 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.234.254:40729 (local address 127.6.234.195:51291)
0620 11:42:43.604884 (+ 179us) negotiation.cc:107] Waiting for socket to connect
0620 11:42:43.604918 (+ 34us) client_negotiation.cc:174] Beginning negotiation
0620 11:42:43.605476 (+ 558us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 11:42:43.605737 (+ 261us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 11:42:43.605741 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 11:42:43.606325 (+ 584us) client_negotiation.cc:189] Negotiated authn=SASL
0620 11:42:43.606513 (+ 188us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 11:42:43.606518 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 11:42:43.607200 (+ 682us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 11:42:43.607201 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 11:42:43.608124 (+ 923us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 11:42:43.608126 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 11:42:43.608503 (+ 377us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 11:42:43.608504 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 11:42:43.608636 (+ 132us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 11:42:43.609141 (+ 505us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 11:42:43.609149 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0620 11:42:43.610755 (+ 1606us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 11:42:43.612359 (+ 1604us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 11:42:43.612365 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 11:42:43.612367 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0620 11:42:43.612585 (+ 218us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 11:42:43.612801 (+ 216us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 11:42:43.612802 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 11:42:43.612802 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0620 11:42:43.612908 (+ 106us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 11:42:43.613169 (+ 261us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 11:42:43.613175 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 11:42:43.613314 (+ 139us) client_negotiation.cc:769] Sending connection context
0620 11:42:43.613423 (+ 109us) client_negotiation.cc:240] Negotiation successful
0620 11:42:43.613574 (+ 151us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":134,"thread_start_us":90,"threads_started":1}
I20260620 11:42:43.614297 10592 heartbeater.cc:347] Connected to a master server at 127.6.234.254:40729
I20260620 11:42:43.614395 10592 heartbeater.cc:464] Registering TS with master...
I20260620 11:42:43.614531 10592 heartbeater.cc:511] Master 127.6.234.254:40729 requested a full tablet report, sending...
I20260620 11:42:43.614871 10133 ts_manager.cc:194] Registered new tserver with Master: cdac5698e97345f1b6ae236b67733273 (127.6.234.195:46335)
I20260620 11:42:43.615237 10133 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.6.234.195@KRBTEST.COM'} at 127.6.234.195:51291
I20260620 11:42:43.621948 7083 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
Jun 20 11:42:43 dist-test-slave-cjc6 krb5kdc[10088](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781955763, etypes {rep=17 tkt=17 ses=17}, joe-interloper@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for joe-interloper@KRBTEST.COM:
Jun 20 11:42:43 dist-test-slave-cjc6 krb5kdc[10088](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781955763, etypes {rep=17 tkt=17 ses=17}, joe-interloper@KRBTEST.COM for kudu/127.6.234.254@KRBTEST.COM
I20260620 11:42:43.643371 10185 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 11:42:43.636872 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45530 (local address 127.6.234.254:40729)
0620 11:42:43.636930 (+ 58us) server_negotiation.cc:206] Beginning negotiation
0620 11:42:43.636932 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 11:42:43.637014 (+ 82us) server_negotiation.cc:407] Connection header received
0620 11:42:43.637059 (+ 45us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 11:42:43.637060 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 11:42:43.637068 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 11:42:43.637119 (+ 51us) server_negotiation.cc:226] Negotiated authn=SASL
0620 11:42:43.637644 (+ 525us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 11:42:43.638176 (+ 532us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 11:42:43.638908 (+ 732us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 11:42:43.639079 (+ 171us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 11:42:43.639128 (+ 49us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 11:42:43.641224 (+ 2096us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 11:42:43.641225 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 11:42:43.641225 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 11:42:43.641230 (+ 5us) server_negotiation.cc:922] calling sasl_server_start()
0620 11:42:43.642451 (+ 1221us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 11:42:43.642750 (+ 299us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 11:42:43.642751 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 11:42:43.642751 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 11:42:43.642800 (+ 49us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 11:42:43.642970 (+ 170us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 11:42:43.642971 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 11:42:43.642971 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 11:42:43.643104 (+ 133us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 11:42:43.643176 (+ 72us) server_negotiation.cc:1035] Waiting for connection context
0620 11:42:43.643258 (+ 82us) server_negotiation.cc:299] Negotiation successful
0620 11:42:43.643312 (+ 54us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":32}
W20260620 11:42:43.643630 10133 server_base.cc:1116] Unauthorized access attempt to method kudu.master.MasterService.ConnectToMaster from {username='joe-interloper', principal='joe-interloper@KRBTEST.COM'} at 127.0.0.1:45530
I20260620 11:42:43.644079 7083 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task1JRg7N/build/release/bin/kudu with pid 10195
I20260620 11:42:43.648371 7083 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task1JRg7N/build/release/bin/kudu with pid 10329
I20260620 11:42:43.652602 7083 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task1JRg7N/build/release/bin/kudu with pid 10463
I20260620 11:42:43.656769 7083 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task1JRg7N/build/release/bin/kudu with pid 10101
2026-06-20T11:42:43Z chronyd exiting
[ OK ] SecurityITest.TestUnauthorizedClientKerberosCredentials (1832 ms)
[ RUN ] SecurityITest.TestAuthorizedSuperuser
Loading random data
Initializing database '/tmp/dist-test-task1JRg7N/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781955746769051-7083-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 20 11:42:43 dist-test-slave-cjc6 krb5kdc[10604](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 20 11:42:43 dist-test-slave-cjc6 krb5kdc[10604](info): set up 2 sockets
Jun 20 11:42:43 dist-test-slave-cjc6 krb5kdc[10604](info): commencing operation
krb5kdc: starting...
W20260620 11:42:44.697580 7083 mini_kdc.cc:121] Time spent starting KDC: real 1.021s user 0.000s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task1JRg7N/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781955746769051-7083-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task1JRg7N/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781955746769051-7083-0/krb5kdc/test-user.keytab.
Jun 20 11:42:44 dist-test-slave-cjc6 krb5kdc[10604](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781955764, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-20T11:42:44Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-20T11:42:44Z Disabled control of system clock
WARNING: no policy specified for kudu/127.6.234.254@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.6.234.254@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.6.234.254 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task1JRg7N/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781955746769051-7083-0/krb5kdc/kudu_127.6.234.254.keytab.
Entry for principal kudu/127.6.234.254 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task1JRg7N/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781955746769051-7083-0/krb5kdc/kudu_127.6.234.254.keytab.
I20260620 11:42:44.814998 7083 external_mini_cluster.cc:1326] Running /tmp/dist-test-task1JRg7N/build/release/bin/kudu
/tmp/dist-test-task1JRg7N/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task1JRg7N/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781955746769051-7083-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task1JRg7N/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781955746769051-7083-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task1JRg7N/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781955746769051-7083-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task1JRg7N/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781955746769051-7083-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.6.234.254:45525
--webserver_interface=127.6.234.254
--webserver_port=0
--keytab_file=/tmp/dist-test-task1JRg7N/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781955746769051-7083-0/krb5kdc/kudu_127.6.234.254.keytab
--principal=kudu/127.6.234.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.6.234.212:33681
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.6.234.254:45525
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task1JRg7N/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781955746769051-7083-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task1JRg7N/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781955746769051-7083-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task1JRg7N/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781955746769051-7083-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 11:42:44.895835 10617 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 11:42:44.896091 10617 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 11:42:44.896132 10617 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 11:42:44.897393 10617 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 11:42:44.897468 10617 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 11:42:44.897495 10617 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 11:42:44.897521 10617 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 11:42:44.897552 10617 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 11:42:44.899223 10617 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.6.234.212:33681
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task1JRg7N/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781955746769051-7083-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task1JRg7N/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781955746769051-7083-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.6.234.254:45525
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task1JRg7N/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781955746769051-7083-0/krb5kdc/kudu_127.6.234.254.keytab
--principal=kudu/127.6.234.254
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.6.234.254:45525
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task1JRg7N/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781955746769051-7083-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.6.234.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.10617
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task1JRg7N/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781955746769051-7083-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 20 Jun 2026 10:32:37 UTC on ccycloud.quasar-bgyofv.root.comops.site
build id 4457173
I20260620 11:42:44.899518 10617 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 11:42:44.899762 10617 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 11:42:44.904963 10623 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 11:42:44.904960 10622 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 11:42:44.904948 10625 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 11:42:44.904995 10617 server_base.cc:1034] running on GCE node
I20260620 11:42:44.905471 10617 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 11:42:44.905838 10617 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 11:42:44.907063 10617 hybrid_clock.cc:648] HybridClock initialized: now 1781955764907050 us; error 48 us; skew 500 ppm
Jun 20 11:42:44 dist-test-slave-cjc6 krb5kdc[10604](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781955764, etypes {rep=17 tkt=17 ses=17}, kudu/127.6.234.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 11:42:44.909598 10617 init.cc:377] Logged in from keytab as kudu/127.6.234.254@KRBTEST.COM (short username kudu)
I20260620 11:42:44.910120 10617 webserver.cc:460] Webserver started at http://127.6.234.254:40293/ using document root <none> and password file <none>
I20260620 11:42:44.910354 10617 fs_manager.cc:359] Metadata directory not provided
I20260620 11:42:44.910429 10617 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 11:42:44.910562 10617 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 11:42:44.911576 10617 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task1JRg7N/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781955746769051-7083-0/minicluster-data/master-0/data/instance:
uuid: "ddbf893f16d94fdf9a2af5d40358a2bb"
format_stamp: "Formatted at 2026-06-20 11:42:44 on dist-test-slave-cjc6"
server_key: "4669db74fcaba8fc00f420d3e9c04bd4"
server_key_iv: "65e2c2f0f84eb96d43c42f9709165885"
server_key_version: "encryptionkey@0"
I20260620 11:42:44.911929 10617 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task1JRg7N/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781955746769051-7083-0/minicluster-data/master-0/wal/instance:
uuid: "ddbf893f16d94fdf9a2af5d40358a2bb"
format_stamp: "Formatted at 2026-06-20 11:42:44 on dist-test-slave-cjc6"
server_key: "4669db74fcaba8fc00f420d3e9c04bd4"
server_key_iv: "65e2c2f0f84eb96d43c42f9709165885"
server_key_version: "encryptionkey@0"
I20260620 11:42:44.913193 10617 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.003s
I20260620 11:42:44.913921 10632 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 11:42:44.914095 10617 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260620 11:42:44.914192 10617 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task1JRg7N/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781955746769051-7083-0/minicluster-data/master-0/data,/tmp/dist-test-task1JRg7N/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781955746769051-7083-0/minicluster-data/master-0/wal
uuid: "ddbf893f16d94fdf9a2af5d40358a2bb"
format_stamp: "Formatted at 2026-06-20 11:42:44 on dist-test-slave-cjc6"
server_key: "4669db74fcaba8fc00f420d3e9c04bd4"
server_key_iv: "65e2c2f0f84eb96d43c42f9709165885"
server_key_version: "encryptionkey@0"
I20260620 11:42:44.914273 10617 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task1JRg7N/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781955746769051-7083-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task1JRg7N/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781955746769051-7083-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task1JRg7N/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781955746769051-7083-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 11:42:44.920857 10617 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 11:42:44.923071 10617 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 11:42:44.923251 10617 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 11:42:44.927488 10617 rpc_server.cc:307] RPC server started. Bound to: 127.6.234.254:45525
I20260620 11:42:44.927516 10684 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.6.234.254:45525 every 8 connection(s)
I20260620 11:42:44.927992 10617 server_base.cc:1166] Dumped server information to /tmp/dist-test-task1JRg7N/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781955746769051-7083-0/minicluster-data/master-0/data/info.pb
I20260620 11:42:44.928599 10685 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 11:42:44.929049 7083 external_mini_cluster.cc:1388] Started /tmp/dist-test-task1JRg7N/build/release/bin/kudu as pid 10617
I20260620 11:42:44.929158 7083 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task1JRg7N/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781955746769051-7083-0/minicluster-data/master-0/wal/instance
I20260620 11:42:44.929347 7083 external_mini_cluster.cc:1428] Setting key 6c43f15ed68182d62ade0af9c3ea61fe
I20260620 11:42:44.931632 10685 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P ddbf893f16d94fdf9a2af5d40358a2bb: Bootstrap starting.
I20260620 11:42:44.932212 10685 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P ddbf893f16d94fdf9a2af5d40358a2bb: Neither blocks nor log segments found. Creating new log.
I20260620 11:42:44.932497 10685 log.cc:824] T 00000000000000000000000000000000 P ddbf893f16d94fdf9a2af5d40358a2bb: Log is configured to *not* fsync() on all Append() calls
I20260620 11:42:44.933245 10685 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P ddbf893f16d94fdf9a2af5d40358a2bb: No bootstrap required, opened a new log
I20260620 11:42:44.935735 10685 raft_consensus.cc:348] T 00000000000000000000000000000000 P ddbf893f16d94fdf9a2af5d40358a2bb [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ddbf893f16d94fdf9a2af5d40358a2bb" member_type: VOTER last_known_addr { host: "127.6.234.254" port: 45525 } }
I20260620 11:42:44.935920 10685 raft_consensus.cc:374] T 00000000000000000000000000000000 P ddbf893f16d94fdf9a2af5d40358a2bb [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 11:42:44.935958 10685 raft_consensus.cc:729] T 00000000000000000000000000000000 P ddbf893f16d94fdf9a2af5d40358a2bb [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ddbf893f16d94fdf9a2af5d40358a2bb, State: Initialized, Role: FOLLOWER
I20260620 11:42:44.936100 10685 consensus_queue.cc:260] T 00000000000000000000000000000000 P ddbf893f16d94fdf9a2af5d40358a2bb [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ddbf893f16d94fdf9a2af5d40358a2bb" member_type: VOTER last_known_addr { host: "127.6.234.254" port: 45525 } }
I20260620 11:42:44.936180 10685 raft_consensus.cc:388] T 00000000000000000000000000000000 P ddbf893f16d94fdf9a2af5d40358a2bb [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260620 11:42:44.936231 10685 raft_consensus.cc:482] T 00000000000000000000000000000000 P ddbf893f16d94fdf9a2af5d40358a2bb [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260620 11:42:44.936278 10685 raft_consensus.cc:3037] T 00000000000000000000000000000000 P ddbf893f16d94fdf9a2af5d40358a2bb [term 0 FOLLOWER]: Advancing to term 1
Jun 20 11:42:44 dist-test-slave-cjc6 krb5kdc[10604](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781955764, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.6.234.254@KRBTEST.COM
I20260620 11:42:44.936830 10685 raft_consensus.cc:504] T 00000000000000000000000000000000 P ddbf893f16d94fdf9a2af5d40358a2bb [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ddbf893f16d94fdf9a2af5d40358a2bb" member_type: VOTER last_known_addr { host: "127.6.234.254" port: 45525 } }
I20260620 11:42:44.936956 10685 leader_election.cc:302] T 00000000000000000000000000000000 P ddbf893f16d94fdf9a2af5d40358a2bb [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: ddbf893f16d94fdf9a2af5d40358a2bb; no voters:
I20260620 11:42:44.937135 10685 leader_election.cc:288] T 00000000000000000000000000000000 P ddbf893f16d94fdf9a2af5d40358a2bb [CANDIDATE]: Term 1 election: Requested vote from peers
I20260620 11:42:44.937168 10690 raft_consensus.cc:2781] T 00000000000000000000000000000000 P ddbf893f16d94fdf9a2af5d40358a2bb [term 1 FOLLOWER]: Leader election won for term 1
I20260620 11:42:44.937319 10690 raft_consensus.cc:686] T 00000000000000000000000000000000 P ddbf893f16d94fdf9a2af5d40358a2bb [term 1 LEADER]: Becoming Leader. State: Replica: ddbf893f16d94fdf9a2af5d40358a2bb, State: Running, Role: LEADER
I20260620 11:42:44.937392 10685 sys_catalog.cc:564] T 00000000000000000000000000000000 P ddbf893f16d94fdf9a2af5d40358a2bb [sys.catalog]: configured and running, proceeding with master startup.
I20260620 11:42:44.937459 10690 consensus_queue.cc:237] T 00000000000000000000000000000000 P ddbf893f16d94fdf9a2af5d40358a2bb [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ddbf893f16d94fdf9a2af5d40358a2bb" member_type: VOTER last_known_addr { host: "127.6.234.254" port: 45525 } }
I20260620 11:42:44.937891 10690 sys_catalog.cc:455] T 00000000000000000000000000000000 P ddbf893f16d94fdf9a2af5d40358a2bb [sys.catalog]: SysCatalogTable state changed. Reason: New leader ddbf893f16d94fdf9a2af5d40358a2bb. Latest consensus state: current_term: 1 leader_uuid: "ddbf893f16d94fdf9a2af5d40358a2bb" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ddbf893f16d94fdf9a2af5d40358a2bb" member_type: VOTER last_known_addr { host: "127.6.234.254" port: 45525 } } }
I20260620 11:42:44.937991 10690 sys_catalog.cc:458] T 00000000000000000000000000000000 P ddbf893f16d94fdf9a2af5d40358a2bb [sys.catalog]: This master's current role is: LEADER
I20260620 11:42:44.937901 10691 sys_catalog.cc:455] T 00000000000000000000000000000000 P ddbf893f16d94fdf9a2af5d40358a2bb [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "ddbf893f16d94fdf9a2af5d40358a2bb" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ddbf893f16d94fdf9a2af5d40358a2bb" member_type: VOTER last_known_addr { host: "127.6.234.254" port: 45525 } } }
I20260620 11:42:44.938164 10691 sys_catalog.cc:458] T 00000000000000000000000000000000 P ddbf893f16d94fdf9a2af5d40358a2bb [sys.catalog]: This master's current role is: LEADER
W20260620 11:42:44.939221 10705 catalog_manager.cc:1558] T 00000000000000000000000000000000 P ddbf893f16d94fdf9a2af5d40358a2bb: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260620 11:42:44.939294 10705 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260620 11:42:44.939366 10699 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260620 11:42:44.939544 10699 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260620 11:42:44.941076 10688 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 11:42:44.930031 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36736 (local address 127.6.234.254:45525)
0620 11:42:44.930216 (+ 185us) server_negotiation.cc:206] Beginning negotiation
0620 11:42:44.930226 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0620 11:42:44.930258 (+ 32us) server_negotiation.cc:407] Connection header received
0620 11:42:44.930698 (+ 440us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 11:42:44.930705 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 11:42:44.930749 (+ 44us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 11:42:44.930867 (+ 118us) server_negotiation.cc:226] Negotiated authn=SASL
0620 11:42:44.931649 (+ 782us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 11:42:44.932713 (+ 1064us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 11:42:44.933745 (+ 1032us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 11:42:44.934154 (+ 409us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 11:42:44.934235 (+ 81us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 11:42:44.937040 (+ 2805us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 11:42:44.937043 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 11:42:44.937043 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 11:42:44.937053 (+ 10us) server_negotiation.cc:922] calling sasl_server_start()
0620 11:42:44.939224 (+ 2171us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 11:42:44.939726 (+ 502us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 11:42:44.939729 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 11:42:44.939730 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 11:42:44.939826 (+ 96us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 11:42:44.940044 (+ 218us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 11:42:44.940044 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 11:42:44.940045 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 11:42:44.940329 (+ 284us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 11:42:44.940488 (+ 159us) server_negotiation.cc:1035] Waiting for connection context
0620 11:42:44.940745 (+ 257us) server_negotiation.cc:299] Negotiation successful
0620 11:42:44.940877 (+ 132us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":134,"thread_start_us":71,"threads_started":1}
I20260620 11:42:44.946139 10699 catalog_manager.cc:1347] Generated new cluster ID: 35ae33e4ea3f430b9fd4a250c0e5c853
I20260620 11:42:44.946198 10699 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260620 11:42:44.957309 10699 catalog_manager.cc:1370] Generated new certificate authority record
I20260620 11:42:44.957927 10699 catalog_manager.cc:1504] Loading token signing keys...
I20260620 11:42:44.962302 10699 catalog_manager.cc:5868] T 00000000000000000000000000000000 P ddbf893f16d94fdf9a2af5d40358a2bb: Generated new TSK 0
I20260620 11:42:44.962491 10699 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.6.234.193@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.6.234.193@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.6.234.193 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task1JRg7N/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781955746769051-7083-0/krb5kdc/kudu_127.6.234.193.keytab.
Entry for principal kudu/127.6.234.193 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task1JRg7N/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781955746769051-7083-0/krb5kdc/kudu_127.6.234.193.keytab.
I20260620 11:42:44.991840 7083 external_mini_cluster.cc:1326] Running /tmp/dist-test-task1JRg7N/build/release/bin/kudu
/tmp/dist-test-task1JRg7N/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task1JRg7N/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781955746769051-7083-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-task1JRg7N/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781955746769051-7083-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task1JRg7N/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781955746769051-7083-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-task1JRg7N/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781955746769051-7083-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.6.234.193:0
--local_ip_for_outbound_sockets=127.6.234.193
--webserver_interface=127.6.234.193
--webserver_port=0
--tserver_master_addrs=127.6.234.254:45525
--keytab_file=/tmp/dist-test-task1JRg7N/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781955746769051-7083-0/krb5kdc/kudu_127.6.234.193.keytab
--principal=kudu/127.6.234.193
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.6.234.212:33681
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task1JRg7N/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781955746769051-7083-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task1JRg7N/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781955746769051-7083-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task1JRg7N/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781955746769051-7083-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 11:42:45.088759 10711 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 11:42:45.088985 10711 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 11:42:45.089008 10711 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 11:42:45.090256 10711 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 11:42:45.090301 10711 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 11:42:45.090327 10711 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.6.234.193
I20260620 11:42:45.091816 10711 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.6.234.212:33681
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task1JRg7N/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781955746769051-7083-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-task1JRg7N/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781955746769051-7083-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task1JRg7N/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781955746769051-7083-0/krb5kdc/kudu_127.6.234.193.keytab
--principal=kudu/127.6.234.193
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.6.234.193:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task1JRg7N/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781955746769051-7083-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.6.234.193
--webserver_port=0
--tserver_master_addrs=127.6.234.254:45525
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.10711
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.6.234.193
--log_dir=/tmp/dist-test-task1JRg7N/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781955746769051-7083-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 20 Jun 2026 10:32:37 UTC on ccycloud.quasar-bgyofv.root.comops.site
build id 4457173
I20260620 11:42:45.092060 10711 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 11:42:45.092301 10711 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 11:42:45.097255 10717 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 11:42:45.097307 10716 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 11:42:45.097453 10711 server_base.cc:1034] running on GCE node
W20260620 11:42:45.097496 10719 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 11:42:45.097800 10711 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 11:42:45.098057 10711 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 11:42:45.099206 10711 hybrid_clock.cc:648] HybridClock initialized: now 1781955765099186 us; error 27 us; skew 500 ppm
Jun 20 11:42:45 dist-test-slave-cjc6 krb5kdc[10604](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781955765, etypes {rep=17 tkt=17 ses=17}, kudu/127.6.234.193@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 11:42:45.102864 10711 init.cc:377] Logged in from keytab as kudu/127.6.234.193@KRBTEST.COM (short username kudu)
I20260620 11:42:45.103379 10711 webserver.cc:460] Webserver started at http://127.6.234.193:46055/ using document root <none> and password file <none>
I20260620 11:42:45.103680 10711 fs_manager.cc:359] Metadata directory not provided
I20260620 11:42:45.103741 10711 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 11:42:45.103883 10711 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 11:42:45.105247 10711 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task1JRg7N/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781955746769051-7083-0/minicluster-data/ts-0/data/instance:
uuid: "273888861e144ea48cb49aea9f5131ac"
format_stamp: "Formatted at 2026-06-20 11:42:45 on dist-test-slave-cjc6"
server_key: "efb9cc0ba64cfccc37ec1424564ea9c9"
server_key_iv: "98e9e72eadd3c5cfae75ddca1b2646e5"
server_key_version: "encryptionkey@0"
I20260620 11:42:45.105615 10711 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task1JRg7N/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781955746769051-7083-0/minicluster-data/ts-0/wal/instance:
uuid: "273888861e144ea48cb49aea9f5131ac"
format_stamp: "Formatted at 2026-06-20 11:42:45 on dist-test-slave-cjc6"
server_key: "efb9cc0ba64cfccc37ec1424564ea9c9"
server_key_iv: "98e9e72eadd3c5cfae75ddca1b2646e5"
server_key_version: "encryptionkey@0"
I20260620 11:42:45.107225 10711 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.001s
I20260620 11:42:45.107924 10726 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 11:42:45.108086 10711 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260620 11:42:45.108181 10711 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task1JRg7N/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781955746769051-7083-0/minicluster-data/ts-0/data,/tmp/dist-test-task1JRg7N/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781955746769051-7083-0/minicluster-data/ts-0/wal
uuid: "273888861e144ea48cb49aea9f5131ac"
format_stamp: "Formatted at 2026-06-20 11:42:45 on dist-test-slave-cjc6"
server_key: "efb9cc0ba64cfccc37ec1424564ea9c9"
server_key_iv: "98e9e72eadd3c5cfae75ddca1b2646e5"
server_key_version: "encryptionkey@0"
I20260620 11:42:45.108260 10711 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task1JRg7N/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781955746769051-7083-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-task1JRg7N/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781955746769051-7083-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-task1JRg7N/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781955746769051-7083-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 11:42:45.127564 10711 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 11:42:45.129339 10711 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 11:42:45.129503 10711 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 11:42:45.129747 10711 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 11:42:45.130062 10711 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 11:42:45.130118 10711 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 11:42:45.130158 10711 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 11:42:45.130205 10711 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 11:42:45.136054 10711 rpc_server.cc:307] RPC server started. Bound to: 127.6.234.193:40711
I20260620 11:42:45.136106 10839 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.6.234.193:40711 every 8 connection(s)
I20260620 11:42:45.136445 10711 server_base.cc:1166] Dumped server information to /tmp/dist-test-task1JRg7N/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781955746769051-7083-0/minicluster-data/ts-0/data/info.pb
I20260620 11:42:45.137521 7083 external_mini_cluster.cc:1388] Started /tmp/dist-test-task1JRg7N/build/release/bin/kudu as pid 10711
I20260620 11:42:45.137624 7083 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task1JRg7N/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781955746769051-7083-0/minicluster-data/ts-0/wal/instance
I20260620 11:42:45.137845 7083 external_mini_cluster.cc:1428] Setting key c593e6218c66d6e61dc63e0e7c6483e3
Jun 20 11:42:45 dist-test-slave-cjc6 krb5kdc[10604](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781955765, etypes {rep=17 tkt=17 ses=17}, kudu/127.6.234.193@KRBTEST.COM for kudu/127.6.234.254@KRBTEST.COM
WARNING: no policy specified for kudu/127.6.234.194@KRBTEST.COM; defaulting to no policy
I20260620 11:42:45.147535 10688 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 11:42:45.137180 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.6.234.193:42857 (local address 127.6.234.254:45525)
0620 11:42:45.137242 (+ 62us) server_negotiation.cc:206] Beginning negotiation
0620 11:42:45.137243 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0620 11:42:45.137478 (+ 235us) server_negotiation.cc:407] Connection header received
0620 11:42:45.138871 (+ 1393us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 11:42:45.138872 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 11:42:45.138883 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 11:42:45.138943 (+ 60us) server_negotiation.cc:226] Negotiated authn=SASL
0620 11:42:45.139796 (+ 853us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 11:42:45.140293 (+ 497us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 11:42:45.141920 (+ 1627us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 11:42:45.142117 (+ 197us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 11:42:45.142174 (+ 57us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 11:42:45.144610 (+ 2436us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 11:42:45.144612 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 11:42:45.144613 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 11:42:45.144620 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0620 11:42:45.146143 (+ 1523us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 11:42:45.146560 (+ 417us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 11:42:45.146562 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 11:42:45.146562 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 11:42:45.146628 (+ 66us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 11:42:45.146867 (+ 239us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 11:42:45.146868 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 11:42:45.146868 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 11:42:45.146990 (+ 122us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 11:42:45.147069 (+ 79us) server_negotiation.cc:1035] Waiting for connection context
0620 11:42:45.147338 (+ 269us) server_negotiation.cc:299] Negotiation successful
0620 11:42:45.147442 (+ 104us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":37}
I20260620 11:42:45.147706 10842 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 11:42:45.137171 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.234.254:45525 (local address 127.6.234.193:42857)
0620 11:42:45.137311 (+ 140us) negotiation.cc:107] Waiting for socket to connect
0620 11:42:45.137372 (+ 61us) client_negotiation.cc:174] Beginning negotiation
0620 11:42:45.138755 (+ 1383us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 11:42:45.139018 (+ 263us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 11:42:45.139023 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 11:42:45.139419 (+ 396us) client_negotiation.cc:189] Negotiated authn=SASL
0620 11:42:45.139697 (+ 278us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 11:42:45.139703 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 11:42:45.140389 (+ 686us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 11:42:45.140390 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 11:42:45.141832 (+ 1442us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 11:42:45.141836 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 11:42:45.142206 (+ 370us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 11:42:45.142207 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 11:42:45.142372 (+ 165us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 11:42:45.142886 (+ 514us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 11:42:45.142896 (+ 10us) client_negotiation.cc:656] calling sasl_client_start()
0620 11:42:45.144512 (+ 1616us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 11:42:45.146229 (+ 1717us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 11:42:45.146236 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 11:42:45.146239 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0620 11:42:45.146466 (+ 227us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 11:42:45.146682 (+ 216us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 11:42:45.146683 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 11:42:45.146684 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0620 11:42:45.146797 (+ 113us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 11:42:45.147061 (+ 264us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 11:42:45.147068 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 11:42:45.147208 (+ 140us) client_negotiation.cc:769] Sending connection context
0620 11:42:45.147318 (+ 110us) client_negotiation.cc:240] Negotiation successful
0620 11:42:45.147473 (+ 155us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":105,"thread_start_us":93,"threads_started":1}
I20260620 11:42:45.148263 10840 heartbeater.cc:347] Connected to a master server at 127.6.234.254:45525
I20260620 11:42:45.148548 10840 heartbeater.cc:464] Registering TS with master...
I20260620 11:42:45.148770 10840 heartbeater.cc:511] Master 127.6.234.254:45525 requested a full tablet report, sending...
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.6.234.194@KRBTEST.COM" created.
I20260620 11:42:45.149358 10649 ts_manager.cc:194] Registered new tserver with Master: 273888861e144ea48cb49aea9f5131ac (127.6.234.193:40711)
I20260620 11:42:45.150068 10649 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.6.234.193@KRBTEST.COM'} at 127.6.234.193:42857
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.6.234.194 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task1JRg7N/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781955746769051-7083-0/krb5kdc/kudu_127.6.234.194.keytab.
Entry for principal kudu/127.6.234.194 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task1JRg7N/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781955746769051-7083-0/krb5kdc/kudu_127.6.234.194.keytab.
I20260620 11:42:45.163988 7083 external_mini_cluster.cc:1326] Running /tmp/dist-test-task1JRg7N/build/release/bin/kudu
/tmp/dist-test-task1JRg7N/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task1JRg7N/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781955746769051-7083-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-task1JRg7N/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781955746769051-7083-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task1JRg7N/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781955746769051-7083-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-task1JRg7N/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781955746769051-7083-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.6.234.194:0
--local_ip_for_outbound_sockets=127.6.234.194
--webserver_interface=127.6.234.194
--webserver_port=0
--tserver_master_addrs=127.6.234.254:45525
--keytab_file=/tmp/dist-test-task1JRg7N/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781955746769051-7083-0/krb5kdc/kudu_127.6.234.194.keytab
--principal=kudu/127.6.234.194
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.6.234.212:33681
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task1JRg7N/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781955746769051-7083-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task1JRg7N/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781955746769051-7083-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task1JRg7N/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781955746769051-7083-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 11:42:45.244285 10845 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 11:42:45.244484 10845 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 11:42:45.244506 10845 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 11:42:45.245582 10845 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 11:42:45.245616 10845 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 11:42:45.245641 10845 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.6.234.194
I20260620 11:42:45.247141 10845 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.6.234.212:33681
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task1JRg7N/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781955746769051-7083-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-task1JRg7N/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781955746769051-7083-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task1JRg7N/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781955746769051-7083-0/krb5kdc/kudu_127.6.234.194.keytab
--principal=kudu/127.6.234.194
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.6.234.194:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task1JRg7N/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781955746769051-7083-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.6.234.194
--webserver_port=0
--tserver_master_addrs=127.6.234.254:45525
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.10845
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.6.234.194
--log_dir=/tmp/dist-test-task1JRg7N/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781955746769051-7083-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 20 Jun 2026 10:32:37 UTC on ccycloud.quasar-bgyofv.root.comops.site
build id 4457173
I20260620 11:42:45.247323 10845 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 11:42:45.247498 10845 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 11:42:45.252413 10850 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 11:42:45.252425 10853 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 11:42:45.252498 10851 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 11:42:45.252605 10845 server_base.cc:1034] running on GCE node
I20260620 11:42:45.252825 10845 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 11:42:45.253080 10845 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 11:42:45.254228 10845 hybrid_clock.cc:648] HybridClock initialized: now 1781955765254217 us; error 31 us; skew 500 ppm
Jun 20 11:42:45 dist-test-slave-cjc6 krb5kdc[10604](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781955765, etypes {rep=17 tkt=17 ses=17}, kudu/127.6.234.194@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 11:42:45.256547 10845 init.cc:377] Logged in from keytab as kudu/127.6.234.194@KRBTEST.COM (short username kudu)
I20260620 11:42:45.257104 10845 webserver.cc:460] Webserver started at http://127.6.234.194:38313/ using document root <none> and password file <none>
I20260620 11:42:45.257366 10845 fs_manager.cc:359] Metadata directory not provided
I20260620 11:42:45.257434 10845 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 11:42:45.257582 10845 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 11:42:45.258661 10845 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task1JRg7N/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781955746769051-7083-0/minicluster-data/ts-1/data/instance:
uuid: "c4bb9154949849b09d5b59420d61d001"
format_stamp: "Formatted at 2026-06-20 11:42:45 on dist-test-slave-cjc6"
server_key: "cf3824b38398634557eda0e3891500d1"
server_key_iv: "1bc6cd0d72b50b6a8d6921a9a5bf8377"
server_key_version: "encryptionkey@0"
I20260620 11:42:45.259042 10845 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task1JRg7N/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781955746769051-7083-0/minicluster-data/ts-1/wal/instance:
uuid: "c4bb9154949849b09d5b59420d61d001"
format_stamp: "Formatted at 2026-06-20 11:42:45 on dist-test-slave-cjc6"
server_key: "cf3824b38398634557eda0e3891500d1"
server_key_iv: "1bc6cd0d72b50b6a8d6921a9a5bf8377"
server_key_version: "encryptionkey@0"
I20260620 11:42:45.260304 10845 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260620 11:42:45.261020 10860 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 11:42:45.261168 10845 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260620 11:42:45.261246 10845 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task1JRg7N/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781955746769051-7083-0/minicluster-data/ts-1/data,/tmp/dist-test-task1JRg7N/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781955746769051-7083-0/minicluster-data/ts-1/wal
uuid: "c4bb9154949849b09d5b59420d61d001"
format_stamp: "Formatted at 2026-06-20 11:42:45 on dist-test-slave-cjc6"
server_key: "cf3824b38398634557eda0e3891500d1"
server_key_iv: "1bc6cd0d72b50b6a8d6921a9a5bf8377"
server_key_version: "encryptionkey@0"
I20260620 11:42:45.261297 10845 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task1JRg7N/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781955746769051-7083-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-task1JRg7N/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781955746769051-7083-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-task1JRg7N/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781955746769051-7083-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 11:42:45.271772 10845 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 11:42:45.273458 10845 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 11:42:45.273566 10845 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 11:42:45.273825 10845 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 11:42:45.274089 10845 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 11:42:45.274120 10845 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 11:42:45.274142 10845 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 11:42:45.274154 10845 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 11:42:45.279911 10845 rpc_server.cc:307] RPC server started. Bound to: 127.6.234.194:41253
I20260620 11:42:45.279983 10973 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.6.234.194:41253 every 8 connection(s)
I20260620 11:42:45.280267 10845 server_base.cc:1166] Dumped server information to /tmp/dist-test-task1JRg7N/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781955746769051-7083-0/minicluster-data/ts-1/data/info.pb
Jun 20 11:42:45 dist-test-slave-cjc6 krb5kdc[10604](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781955765, etypes {rep=17 tkt=17 ses=17}, kudu/127.6.234.194@KRBTEST.COM for kudu/127.6.234.254@KRBTEST.COM
I20260620 11:42:45.288697 7083 external_mini_cluster.cc:1388] Started /tmp/dist-test-task1JRg7N/build/release/bin/kudu as pid 10845
I20260620 11:42:45.288818 7083 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task1JRg7N/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781955746769051-7083-0/minicluster-data/ts-1/wal/instance
I20260620 11:42:45.288928 10688 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 11:42:45.281005 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.6.234.194:56817 (local address 127.6.234.254:45525)
0620 11:42:45.281078 (+ 73us) server_negotiation.cc:206] Beginning negotiation
0620 11:42:45.281080 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 11:42:45.281257 (+ 177us) server_negotiation.cc:407] Connection header received
0620 11:42:45.282193 (+ 936us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 11:42:45.282195 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 11:42:45.282205 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 11:42:45.282263 (+ 58us) server_negotiation.cc:226] Negotiated authn=SASL
0620 11:42:45.282813 (+ 550us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 11:42:45.283293 (+ 480us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 11:42:45.284225 (+ 932us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 11:42:45.284408 (+ 183us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 11:42:45.284459 (+ 51us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 11:42:45.286571 (+ 2112us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 11:42:45.286573 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 11:42:45.286573 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 11:42:45.286579 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0620 11:42:45.287733 (+ 1154us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 11:42:45.288085 (+ 352us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 11:42:45.288086 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 11:42:45.288086 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 11:42:45.288132 (+ 46us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 11:42:45.288330 (+ 198us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 11:42:45.288331 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 11:42:45.288331 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 11:42:45.288435 (+ 104us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 11:42:45.288483 (+ 48us) server_negotiation.cc:1035] Waiting for connection context
0620 11:42:45.288739 (+ 256us) server_negotiation.cc:299] Negotiation successful
0620 11:42:45.288826 (+ 87us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":47}
I20260620 11:42:45.289039 7083 external_mini_cluster.cc:1428] Setting key e5120e99a9b2496f7dc78ac9a33f2afb
I20260620 11:42:45.289083 10976 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 11:42:45.280975 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.234.254:45525 (local address 127.6.234.194:56817)
0620 11:42:45.281130 (+ 155us) negotiation.cc:107] Waiting for socket to connect
0620 11:42:45.281169 (+ 39us) client_negotiation.cc:174] Beginning negotiation
0620 11:42:45.282098 (+ 929us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 11:42:45.282318 (+ 220us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 11:42:45.282322 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 11:42:45.282565 (+ 243us) client_negotiation.cc:189] Negotiated authn=SASL
0620 11:42:45.282745 (+ 180us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 11:42:45.282748 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 11:42:45.283369 (+ 621us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 11:42:45.283370 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 11:42:45.284154 (+ 784us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 11:42:45.284155 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 11:42:45.284479 (+ 324us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 11:42:45.284480 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 11:42:45.284580 (+ 100us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 11:42:45.285107 (+ 527us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 11:42:45.285115 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0620 11:42:45.286487 (+ 1372us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 11:42:45.287802 (+ 1315us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 11:42:45.287807 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 11:42:45.287809 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0620 11:42:45.288022 (+ 213us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 11:42:45.288189 (+ 167us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 11:42:45.288189 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 11:42:45.288189 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0620 11:42:45.288278 (+ 89us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 11:42:45.288488 (+ 210us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 11:42:45.288491 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 11:42:45.288609 (+ 118us) client_negotiation.cc:769] Sending connection context
0620 11:42:45.288687 (+ 78us) client_negotiation.cc:240] Negotiation successful
0620 11:42:45.288839 (+ 152us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":114,"thread_start_us":85,"threads_started":1}
I20260620 11:42:45.289543 10974 heartbeater.cc:347] Connected to a master server at 127.6.234.254:45525
I20260620 11:42:45.289657 10974 heartbeater.cc:464] Registering TS with master...
I20260620 11:42:45.289848 10974 heartbeater.cc:511] Master 127.6.234.254:45525 requested a full tablet report, sending...
I20260620 11:42:45.290256 10649 ts_manager.cc:194] Registered new tserver with Master: c4bb9154949849b09d5b59420d61d001 (127.6.234.194:41253)
I20260620 11:42:45.290768 10649 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.6.234.194@KRBTEST.COM'} at 127.6.234.194:56817
WARNING: no policy specified for kudu/127.6.234.195@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.6.234.195@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.6.234.195 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task1JRg7N/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781955746769051-7083-0/krb5kdc/kudu_127.6.234.195.keytab.
Entry for principal kudu/127.6.234.195 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task1JRg7N/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781955746769051-7083-0/krb5kdc/kudu_127.6.234.195.keytab.
I20260620 11:42:45.312296 7083 external_mini_cluster.cc:1326] Running /tmp/dist-test-task1JRg7N/build/release/bin/kudu
/tmp/dist-test-task1JRg7N/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task1JRg7N/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781955746769051-7083-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-task1JRg7N/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781955746769051-7083-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task1JRg7N/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781955746769051-7083-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-task1JRg7N/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781955746769051-7083-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.6.234.195:0
--local_ip_for_outbound_sockets=127.6.234.195
--webserver_interface=127.6.234.195
--webserver_port=0
--tserver_master_addrs=127.6.234.254:45525
--keytab_file=/tmp/dist-test-task1JRg7N/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781955746769051-7083-0/krb5kdc/kudu_127.6.234.195.keytab
--principal=kudu/127.6.234.195
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.6.234.212:33681
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task1JRg7N/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781955746769051-7083-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task1JRg7N/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781955746769051-7083-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task1JRg7N/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781955746769051-7083-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 11:42:45.391762 10979 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 11:42:45.391999 10979 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 11:42:45.392037 10979 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 11:42:45.393227 10979 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 11:42:45.393298 10979 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 11:42:45.393342 10979 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.6.234.195
I20260620 11:42:45.394876 10979 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.6.234.212:33681
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task1JRg7N/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781955746769051-7083-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-task1JRg7N/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781955746769051-7083-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task1JRg7N/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781955746769051-7083-0/krb5kdc/kudu_127.6.234.195.keytab
--principal=kudu/127.6.234.195
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.6.234.195:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task1JRg7N/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781955746769051-7083-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.6.234.195
--webserver_port=0
--tserver_master_addrs=127.6.234.254:45525
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.10979
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.6.234.195
--log_dir=/tmp/dist-test-task1JRg7N/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781955746769051-7083-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 20 Jun 2026 10:32:37 UTC on ccycloud.quasar-bgyofv.root.comops.site
build id 4457173
I20260620 11:42:45.395148 10979 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 11:42:45.395370 10979 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 11:42:45.400359 10984 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 11:42:45.400363 10985 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 11:42:45.400369 10987 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 11:42:45.400609 10979 server_base.cc:1034] running on GCE node
I20260620 11:42:45.400862 10979 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 11:42:45.401111 10979 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 11:42:45.402268 10979 hybrid_clock.cc:648] HybridClock initialized: now 1781955765402259 us; error 39 us; skew 500 ppm
Jun 20 11:42:45 dist-test-slave-cjc6 krb5kdc[10604](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781955765, etypes {rep=17 tkt=17 ses=17}, kudu/127.6.234.195@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 11:42:45.404650 10979 init.cc:377] Logged in from keytab as kudu/127.6.234.195@KRBTEST.COM (short username kudu)
I20260620 11:42:45.405050 10979 webserver.cc:460] Webserver started at http://127.6.234.195:46169/ using document root <none> and password file <none>
I20260620 11:42:45.405236 10979 fs_manager.cc:359] Metadata directory not provided
I20260620 11:42:45.405275 10979 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 11:42:45.405361 10979 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 11:42:45.406436 10979 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task1JRg7N/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781955746769051-7083-0/minicluster-data/ts-2/data/instance:
uuid: "700ab1c1ac03432b8c54b223dff57faf"
format_stamp: "Formatted at 2026-06-20 11:42:45 on dist-test-slave-cjc6"
server_key: "04c6a87c2f0e307e401102f7ea408a2a"
server_key_iv: "e2bb1513aa946c74bc143a05236db74b"
server_key_version: "encryptionkey@0"
I20260620 11:42:45.406724 10979 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task1JRg7N/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781955746769051-7083-0/minicluster-data/ts-2/wal/instance:
uuid: "700ab1c1ac03432b8c54b223dff57faf"
format_stamp: "Formatted at 2026-06-20 11:42:45 on dist-test-slave-cjc6"
server_key: "04c6a87c2f0e307e401102f7ea408a2a"
server_key_iv: "e2bb1513aa946c74bc143a05236db74b"
server_key_version: "encryptionkey@0"
I20260620 11:42:45.407913 10979 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260620 11:42:45.408547 10994 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 11:42:45.408736 10979 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260620 11:42:45.408818 10979 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task1JRg7N/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781955746769051-7083-0/minicluster-data/ts-2/data,/tmp/dist-test-task1JRg7N/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781955746769051-7083-0/minicluster-data/ts-2/wal
uuid: "700ab1c1ac03432b8c54b223dff57faf"
format_stamp: "Formatted at 2026-06-20 11:42:45 on dist-test-slave-cjc6"
server_key: "04c6a87c2f0e307e401102f7ea408a2a"
server_key_iv: "e2bb1513aa946c74bc143a05236db74b"
server_key_version: "encryptionkey@0"
I20260620 11:42:45.408870 10979 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task1JRg7N/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781955746769051-7083-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-task1JRg7N/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781955746769051-7083-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-task1JRg7N/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781955746769051-7083-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 11:42:45.416716 10979 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 11:42:45.418929 10979 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 11:42:45.419032 10979 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 11:42:45.419188 10979 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 11:42:45.419411 10979 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 11:42:45.419440 10979 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 11:42:45.419461 10979 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 11:42:45.419481 10979 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 11:42:45.424921 10979 rpc_server.cc:307] RPC server started. Bound to: 127.6.234.195:44143
I20260620 11:42:45.424991 11107 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.6.234.195:44143 every 8 connection(s)
I20260620 11:42:45.425284 10979 server_base.cc:1166] Dumped server information to /tmp/dist-test-task1JRg7N/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781955746769051-7083-0/minicluster-data/ts-2/data/info.pb
I20260620 11:42:45.426610 7083 external_mini_cluster.cc:1388] Started /tmp/dist-test-task1JRg7N/build/release/bin/kudu as pid 10979
I20260620 11:42:45.426733 7083 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task1JRg7N/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781955746769051-7083-0/minicluster-data/ts-2/wal/instance
I20260620 11:42:45.426919 7083 external_mini_cluster.cc:1428] Setting key 2eec825605241a546a3b28ddc06aa000
Jun 20 11:42:45 dist-test-slave-cjc6 krb5kdc[10604](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781955765, etypes {rep=17 tkt=17 ses=17}, kudu/127.6.234.195@KRBTEST.COM for kudu/127.6.234.254@KRBTEST.COM
I20260620 11:42:45.434088 10688 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 11:42:45.426019 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.6.234.195:58539 (local address 127.6.234.254:45525)
0620 11:42:45.426063 (+ 44us) server_negotiation.cc:206] Beginning negotiation
0620 11:42:45.426065 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 11:42:45.426275 (+ 210us) server_negotiation.cc:407] Connection header received
0620 11:42:45.427233 (+ 958us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 11:42:45.427234 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 11:42:45.427245 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 11:42:45.427314 (+ 69us) server_negotiation.cc:226] Negotiated authn=SASL
0620 11:42:45.427906 (+ 592us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 11:42:45.428414 (+ 508us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 11:42:45.429395 (+ 981us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 11:42:45.429576 (+ 181us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 11:42:45.429631 (+ 55us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 11:42:45.431768 (+ 2137us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 11:42:45.431769 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 11:42:45.431770 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 11:42:45.431776 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0620 11:42:45.432890 (+ 1114us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 11:42:45.433297 (+ 407us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 11:42:45.433298 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 11:42:45.433299 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 11:42:45.433355 (+ 56us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 11:42:45.433538 (+ 183us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 11:42:45.433539 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 11:42:45.433539 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 11:42:45.433634 (+ 95us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 11:42:45.433683 (+ 49us) server_negotiation.cc:1035] Waiting for connection context
0620 11:42:45.433968 (+ 285us) server_negotiation.cc:299] Negotiation successful
0620 11:42:45.434027 (+ 59us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":20}
I20260620 11:42:45.434303 11110 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 11:42:45.425988 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.234.254:45525 (local address 127.6.234.195:58539)
0620 11:42:45.426145 (+ 157us) negotiation.cc:107] Waiting for socket to connect
0620 11:42:45.426185 (+ 40us) client_negotiation.cc:174] Beginning negotiation
0620 11:42:45.427154 (+ 969us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 11:42:45.427364 (+ 210us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 11:42:45.427367 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 11:42:45.427661 (+ 294us) client_negotiation.cc:189] Negotiated authn=SASL
0620 11:42:45.427843 (+ 182us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 11:42:45.427848 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 11:42:45.428474 (+ 626us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 11:42:45.428475 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 11:42:45.429323 (+ 848us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 11:42:45.429325 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 11:42:45.429650 (+ 325us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 11:42:45.429650 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 11:42:45.429802 (+ 152us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 11:42:45.430332 (+ 530us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 11:42:45.430340 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0620 11:42:45.431688 (+ 1348us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 11:42:45.432975 (+ 1287us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 11:42:45.432980 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 11:42:45.432982 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0620 11:42:45.433226 (+ 244us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 11:42:45.433399 (+ 173us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 11:42:45.433399 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 11:42:45.433400 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0620 11:42:45.433488 (+ 88us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 11:42:45.433674 (+ 186us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 11:42:45.433676 (+ 2us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 11:42:45.433858 (+ 182us) client_negotiation.cc:769] Sending connection context
0620 11:42:45.433942 (+ 84us) client_negotiation.cc:240] Negotiation successful
0620 11:42:45.434054 (+ 112us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":117,"thread_start_us":88,"threads_started":1}
I20260620 11:42:45.434823 11108 heartbeater.cc:347] Connected to a master server at 127.6.234.254:45525
I20260620 11:42:45.434916 11108 heartbeater.cc:464] Registering TS with master...
I20260620 11:42:45.435099 11108 heartbeater.cc:511] Master 127.6.234.254:45525 requested a full tablet report, sending...
I20260620 11:42:45.435508 10649 ts_manager.cc:194] Registered new tserver with Master: 700ab1c1ac03432b8c54b223dff57faf (127.6.234.195:44143)
I20260620 11:42:45.435866 10649 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.6.234.195@KRBTEST.COM'} at 127.6.234.195:58539
I20260620 11:42:45.438613 7083 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
Jun 20 11:42:45 dist-test-slave-cjc6 krb5kdc[10604](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781955765, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
Jun 20 11:42:45 dist-test-slave-cjc6 krb5kdc[10604](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781955765, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.6.234.193@KRBTEST.COM
I20260620 11:42:45.460429 11115 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 11:42:45.453267 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40698 (local address 127.6.234.193:40711)
0620 11:42:45.453420 (+ 153us) server_negotiation.cc:206] Beginning negotiation
0620 11:42:45.453427 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0620 11:42:45.453443 (+ 16us) server_negotiation.cc:407] Connection header received
0620 11:42:45.453469 (+ 26us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 11:42:45.453471 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 11:42:45.453485 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 11:42:45.453557 (+ 72us) server_negotiation.cc:226] Negotiated authn=SASL
0620 11:42:45.454181 (+ 624us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 11:42:45.454914 (+ 733us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 11:42:45.455600 (+ 686us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 11:42:45.455917 (+ 317us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 11:42:45.455971 (+ 54us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 11:42:45.458084 (+ 2113us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 11:42:45.458094 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 11:42:45.458094 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 11:42:45.458101 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0620 11:42:45.459487 (+ 1386us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 11:42:45.459783 (+ 296us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 11:42:45.459786 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 11:42:45.459786 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 11:42:45.459839 (+ 53us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 11:42:45.460013 (+ 174us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 11:42:45.460013 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 11:42:45.460014 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 11:42:45.460132 (+ 118us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 11:42:45.460237 (+ 105us) server_negotiation.cc:1035] Waiting for connection context
0620 11:42:45.460291 (+ 54us) server_negotiation.cc:299] Negotiation successful
0620 11:42:45.460350 (+ 59us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":116,"thread_start_us":94,"threads_started":1}
Jun 20 11:42:45 dist-test-slave-cjc6 krb5kdc[10604](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781955765, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.6.234.254@KRBTEST.COM
I20260620 11:42:45.467834 10688 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 11:42:45.461675 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36740 (local address 127.6.234.254:45525)
0620 11:42:45.461749 (+ 74us) server_negotiation.cc:206] Beginning negotiation
0620 11:42:45.461751 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 11:42:45.461759 (+ 8us) server_negotiation.cc:407] Connection header received
0620 11:42:45.461809 (+ 50us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 11:42:45.461810 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 11:42:45.461817 (+ 7us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 11:42:45.461869 (+ 52us) server_negotiation.cc:226] Negotiated authn=SASL
0620 11:42:45.462410 (+ 541us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 11:42:45.462923 (+ 513us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 11:42:45.463552 (+ 629us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 11:42:45.463769 (+ 217us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 11:42:45.463825 (+ 56us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 11:42:45.465860 (+ 2035us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 11:42:45.465861 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 11:42:45.465861 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 11:42:45.465867 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0620 11:42:45.466976 (+ 1109us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 11:42:45.467258 (+ 282us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 11:42:45.467259 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 11:42:45.467259 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 11:42:45.467304 (+ 45us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 11:42:45.467443 (+ 139us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 11:42:45.467444 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 11:42:45.467444 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 11:42:45.467568 (+ 124us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 11:42:45.467638 (+ 70us) server_negotiation.cc:1035] Waiting for connection context
0620 11:42:45.467719 (+ 81us) server_negotiation.cc:299] Negotiation successful
0620 11:42:45.467773 (+ 54us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":46}
W20260620 11:42:45.468075 10649 server_base.cc:1116] Unauthorized access attempt to method kudu.master.MasterService.TSHeartbeat from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:36740
I20260620 11:42:45.468421 7083 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task1JRg7N/build/release/bin/kudu with pid 10711
I20260620 11:42:45.473204 7083 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task1JRg7N/build/release/bin/kudu with pid 10845
I20260620 11:42:45.477487 7083 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task1JRg7N/build/release/bin/kudu with pid 10979
I20260620 11:42:45.481552 7083 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task1JRg7N/build/release/bin/kudu with pid 10617
2026-06-20T11:42:45Z chronyd exiting
[ OK ] SecurityITest.TestAuthorizedSuperuser (1824 ms)
[ RUN ] SecurityITest.TestDisableWebUI
Loading random data
Initializing database '/tmp/dist-test-task1JRg7N/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781955746769051-7083-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 20 11:42:45 dist-test-slave-cjc6 krb5kdc[11120](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 20 11:42:45 dist-test-slave-cjc6 krb5kdc[11120](info): set up 2 sockets
Jun 20 11:42:45 dist-test-slave-cjc6 krb5kdc[11120](info): commencing operation
krb5kdc: starting...
W20260620 11:42:46.569649 7083 mini_kdc.cc:121] Time spent starting KDC: real 1.068s user 0.000s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task1JRg7N/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781955746769051-7083-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task1JRg7N/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781955746769051-7083-0/krb5kdc/test-user.keytab.
Jun 20 11:42:46 dist-test-slave-cjc6 krb5kdc[11120](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781955766, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-20T11:42:46Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-20T11:42:46Z Disabled control of system clock
WARNING: no policy specified for kudu/127.6.234.254@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.6.234.254@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.6.234.254 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task1JRg7N/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781955746769051-7083-0/krb5kdc/kudu_127.6.234.254.keytab.
Entry for principal kudu/127.6.234.254 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task1JRg7N/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781955746769051-7083-0/krb5kdc/kudu_127.6.234.254.keytab.
I20260620 11:42:46.684499 7083 external_mini_cluster.cc:1326] Running /tmp/dist-test-task1JRg7N/build/release/bin/kudu
/tmp/dist-test-task1JRg7N/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task1JRg7N/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781955746769051-7083-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task1JRg7N/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781955746769051-7083-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task1JRg7N/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781955746769051-7083-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task1JRg7N/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781955746769051-7083-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.6.234.254:36839
--webserver_interface=127.6.234.254
--webserver_port=0
--keytab_file=/tmp/dist-test-task1JRg7N/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781955746769051-7083-0/krb5kdc/kudu_127.6.234.254.keytab
--principal=kudu/127.6.234.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.6.234.212:35613
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.6.234.254:36839
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--webserver_enabled=0 with env {KRB5CCNAME=/tmp/dist-test-task1JRg7N/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781955746769051-7083-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task1JRg7N/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781955746769051-7083-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task1JRg7N/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781955746769051-7083-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 11:42:46.766027 11133 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 11:42:46.766224 11133 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 11:42:46.766247 11133 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 11:42:46.767418 11133 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 11:42:46.767462 11133 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 11:42:46.767475 11133 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 11:42:46.767527 11133 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 11:42:46.767549 11133 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 11:42:46.769099 11133 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.6.234.212:35613
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task1JRg7N/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781955746769051-7083-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task1JRg7N/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781955746769051-7083-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.6.234.254:36839
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task1JRg7N/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781955746769051-7083-0/krb5kdc/kudu_127.6.234.254.keytab
--principal=kudu/127.6.234.254
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.6.234.254:36839
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--webserver_enabled=false
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task1JRg7N/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781955746769051-7083-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.6.234.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.11133
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task1JRg7N/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781955746769051-7083-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 20 Jun 2026 10:32:37 UTC on ccycloud.quasar-bgyofv.root.comops.site
build id 4457173
I20260620 11:42:46.769344 11133 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 11:42:46.769573 11133 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 11:42:46.774267 11138 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 11:42:46.774282 11139 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 11:42:46.774284 11141 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 11:42:46.774369 11133 server_base.cc:1034] running on GCE node
I20260620 11:42:46.774688 11133 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 11:42:46.774971 11133 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 11:42:46.776113 11133 hybrid_clock.cc:648] HybridClock initialized: now 1781955766776104 us; error 31 us; skew 500 ppm
Jun 20 11:42:46 dist-test-slave-cjc6 krb5kdc[11120](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781955766, etypes {rep=17 tkt=17 ses=17}, kudu/127.6.234.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 11:42:46.778684 11133 init.cc:377] Logged in from keytab as kudu/127.6.234.254@KRBTEST.COM (short username kudu)
I20260620 11:42:46.779083 11133 fs_manager.cc:359] Metadata directory not provided
I20260620 11:42:46.779151 11133 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 11:42:46.779292 11133 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 11:42:46.780311 11133 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task1JRg7N/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781955746769051-7083-0/minicluster-data/master-0/data/instance:
uuid: "46b14fdaa32d4783bb3a63a5574997e5"
format_stamp: "Formatted at 2026-06-20 11:42:46 on dist-test-slave-cjc6"
server_key: "f84b8153fd88e9fa868c952dc18cb52c"
server_key_iv: "7b41e7deaf382cdd418aeb5fb4d23ea9"
server_key_version: "encryptionkey@0"
I20260620 11:42:46.780676 11133 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task1JRg7N/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781955746769051-7083-0/minicluster-data/master-0/wal/instance:
uuid: "46b14fdaa32d4783bb3a63a5574997e5"
format_stamp: "Formatted at 2026-06-20 11:42:46 on dist-test-slave-cjc6"
server_key: "f84b8153fd88e9fa868c952dc18cb52c"
server_key_iv: "7b41e7deaf382cdd418aeb5fb4d23ea9"
server_key_version: "encryptionkey@0"
I20260620 11:42:46.782019 11133 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260620 11:42:46.782735 11147 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 11:42:46.782934 11133 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260620 11:42:46.783031 11133 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task1JRg7N/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781955746769051-7083-0/minicluster-data/master-0/data,/tmp/dist-test-task1JRg7N/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781955746769051-7083-0/minicluster-data/master-0/wal
uuid: "46b14fdaa32d4783bb3a63a5574997e5"
format_stamp: "Formatted at 2026-06-20 11:42:46 on dist-test-slave-cjc6"
server_key: "f84b8153fd88e9fa868c952dc18cb52c"
server_key_iv: "7b41e7deaf382cdd418aeb5fb4d23ea9"
server_key_version: "encryptionkey@0"
I20260620 11:42:46.783115 11133 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task1JRg7N/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781955746769051-7083-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task1JRg7N/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781955746769051-7083-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task1JRg7N/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781955746769051-7083-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 11:42:46.806536 11133 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 11:42:46.808962 11133 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 11:42:46.809121 11133 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 11:42:46.813277 11133 rpc_server.cc:307] RPC server started. Bound to: 127.6.234.254:36839
I20260620 11:42:46.813302 11199 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.6.234.254:36839 every 8 connection(s)
I20260620 11:42:46.813596 11133 server_base.cc:1166] Dumped server information to /tmp/dist-test-task1JRg7N/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781955746769051-7083-0/minicluster-data/master-0/data/info.pb
I20260620 11:42:46.814551 11200 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 11:42:46.816815 11200 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 46b14fdaa32d4783bb3a63a5574997e5: Bootstrap starting.
I20260620 11:42:46.817421 11200 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 46b14fdaa32d4783bb3a63a5574997e5: Neither blocks nor log segments found. Creating new log.
I20260620 11:42:46.817741 11200 log.cc:824] T 00000000000000000000000000000000 P 46b14fdaa32d4783bb3a63a5574997e5: Log is configured to *not* fsync() on all Append() calls
I20260620 11:42:46.818403 11200 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 46b14fdaa32d4783bb3a63a5574997e5: No bootstrap required, opened a new log
I20260620 11:42:46.819150 7083 external_mini_cluster.cc:1388] Started /tmp/dist-test-task1JRg7N/build/release/bin/kudu as pid 11133
I20260620 11:42:46.819278 7083 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task1JRg7N/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781955746769051-7083-0/minicluster-data/master-0/wal/instance
I20260620 11:42:46.819447 7083 external_mini_cluster.cc:1428] Setting key d261ab79d7a2c3d0aca6bf07eba69f06
I20260620 11:42:46.820749 11200 raft_consensus.cc:348] T 00000000000000000000000000000000 P 46b14fdaa32d4783bb3a63a5574997e5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "46b14fdaa32d4783bb3a63a5574997e5" member_type: VOTER last_known_addr { host: "127.6.234.254" port: 36839 } }
I20260620 11:42:46.820911 11200 raft_consensus.cc:374] T 00000000000000000000000000000000 P 46b14fdaa32d4783bb3a63a5574997e5 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 11:42:46.820963 11200 raft_consensus.cc:729] T 00000000000000000000000000000000 P 46b14fdaa32d4783bb3a63a5574997e5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 46b14fdaa32d4783bb3a63a5574997e5, State: Initialized, Role: FOLLOWER
I20260620 11:42:46.821107 11200 consensus_queue.cc:260] T 00000000000000000000000000000000 P 46b14fdaa32d4783bb3a63a5574997e5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "46b14fdaa32d4783bb3a63a5574997e5" member_type: VOTER last_known_addr { host: "127.6.234.254" port: 36839 } }
I20260620 11:42:46.821199 11200 raft_consensus.cc:388] T 00000000000000000000000000000000 P 46b14fdaa32d4783bb3a63a5574997e5 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260620 11:42:46.821244 11200 raft_consensus.cc:482] T 00000000000000000000000000000000 P 46b14fdaa32d4783bb3a63a5574997e5 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260620 11:42:46.821305 11200 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 46b14fdaa32d4783bb3a63a5574997e5 [term 0 FOLLOWER]: Advancing to term 1
I20260620 11:42:46.822008 11200 raft_consensus.cc:504] T 00000000000000000000000000000000 P 46b14fdaa32d4783bb3a63a5574997e5 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "46b14fdaa32d4783bb3a63a5574997e5" member_type: VOTER last_known_addr { host: "127.6.234.254" port: 36839 } }
I20260620 11:42:46.822098 11200 leader_election.cc:302] T 00000000000000000000000000000000 P 46b14fdaa32d4783bb3a63a5574997e5 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 46b14fdaa32d4783bb3a63a5574997e5; no voters:
I20260620 11:42:46.822327 11200 leader_election.cc:288] T 00000000000000000000000000000000 P 46b14fdaa32d4783bb3a63a5574997e5 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260620 11:42:46.822381 11205 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 46b14fdaa32d4783bb3a63a5574997e5 [term 1 FOLLOWER]: Leader election won for term 1
I20260620 11:42:46.822506 11205 raft_consensus.cc:686] T 00000000000000000000000000000000 P 46b14fdaa32d4783bb3a63a5574997e5 [term 1 LEADER]: Becoming Leader. State: Replica: 46b14fdaa32d4783bb3a63a5574997e5, State: Running, Role: LEADER
I20260620 11:42:46.822589 11205 consensus_queue.cc:237] T 00000000000000000000000000000000 P 46b14fdaa32d4783bb3a63a5574997e5 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "46b14fdaa32d4783bb3a63a5574997e5" member_type: VOTER last_known_addr { host: "127.6.234.254" port: 36839 } }
I20260620 11:42:46.822595 11200 sys_catalog.cc:564] T 00000000000000000000000000000000 P 46b14fdaa32d4783bb3a63a5574997e5 [sys.catalog]: configured and running, proceeding with master startup.
I20260620 11:42:46.823887 11206 sys_catalog.cc:455] T 00000000000000000000000000000000 P 46b14fdaa32d4783bb3a63a5574997e5 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "46b14fdaa32d4783bb3a63a5574997e5" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "46b14fdaa32d4783bb3a63a5574997e5" member_type: VOTER last_known_addr { host: "127.6.234.254" port: 36839 } } }
I20260620 11:42:46.823987 11206 sys_catalog.cc:458] T 00000000000000000000000000000000 P 46b14fdaa32d4783bb3a63a5574997e5 [sys.catalog]: This master's current role is: LEADER
I20260620 11:42:46.824190 11207 sys_catalog.cc:455] T 00000000000000000000000000000000 P 46b14fdaa32d4783bb3a63a5574997e5 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 46b14fdaa32d4783bb3a63a5574997e5. Latest consensus state: current_term: 1 leader_uuid: "46b14fdaa32d4783bb3a63a5574997e5" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "46b14fdaa32d4783bb3a63a5574997e5" member_type: VOTER last_known_addr { host: "127.6.234.254" port: 36839 } } }
I20260620 11:42:46.824187 11220 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260620 11:42:46.824319 11207 sys_catalog.cc:458] T 00000000000000000000000000000000 P 46b14fdaa32d4783bb3a63a5574997e5 [sys.catalog]: This master's current role is: LEADER
I20260620 11:42:46.824770 11220 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260620 11:42:46.826318 11220 catalog_manager.cc:1347] Generated new cluster ID: 60d61fe51823408bbab67a3bff6dc5e5
I20260620 11:42:46.826375 11220 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
Jun 20 11:42:46 dist-test-slave-cjc6 krb5kdc[11120](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781955766, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.6.234.254@KRBTEST.COM
I20260620 11:42:46.831271 11204 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 11:42:46.820039 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60650 (local address 127.6.234.254:36839)
0620 11:42:46.820223 (+ 184us) server_negotiation.cc:206] Beginning negotiation
0620 11:42:46.820231 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0620 11:42:46.820256 (+ 25us) server_negotiation.cc:407] Connection header received
0620 11:42:46.820730 (+ 474us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 11:42:46.820736 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 11:42:46.820787 (+ 51us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 11:42:46.820972 (+ 185us) server_negotiation.cc:226] Negotiated authn=SASL
0620 11:42:46.821809 (+ 837us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 11:42:46.823112 (+ 1303us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 11:42:46.824043 (+ 931us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 11:42:46.824661 (+ 618us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 11:42:46.824774 (+ 113us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 11:42:46.827549 (+ 2775us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 11:42:46.827553 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 11:42:46.827553 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 11:42:46.827564 (+ 11us) server_negotiation.cc:922] calling sasl_server_start()
0620 11:42:46.829567 (+ 2003us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 11:42:46.830008 (+ 441us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 11:42:46.830010 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 11:42:46.830011 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 11:42:46.830132 (+ 121us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 11:42:46.830361 (+ 229us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 11:42:46.830362 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 11:42:46.830362 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 11:42:46.830639 (+ 277us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 11:42:46.830787 (+ 148us) server_negotiation.cc:1035] Waiting for connection context
0620 11:42:46.830933 (+ 146us) server_negotiation.cc:299] Negotiation successful
0620 11:42:46.831058 (+ 125us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":125,"thread_start_us":85,"threads_started":1}
I20260620 11:42:46.856453 11220 catalog_manager.cc:1370] Generated new certificate authority record
I20260620 11:42:46.857264 11220 catalog_manager.cc:1504] Loading token signing keys...
I20260620 11:42:46.865244 11220 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 46b14fdaa32d4783bb3a63a5574997e5: Generated new TSK 0
I20260620 11:42:46.865466 11220 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.6.234.193@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.6.234.193@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.6.234.193 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task1JRg7N/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781955746769051-7083-0/krb5kdc/kudu_127.6.234.193.keytab.
Entry for principal kudu/127.6.234.193 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task1JRg7N/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781955746769051-7083-0/krb5kdc/kudu_127.6.234.193.keytab.
I20260620 11:42:46.896015 7083 external_mini_cluster.cc:1326] Running /tmp/dist-test-task1JRg7N/build/release/bin/kudu
/tmp/dist-test-task1JRg7N/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task1JRg7N/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781955746769051-7083-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-task1JRg7N/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781955746769051-7083-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task1JRg7N/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781955746769051-7083-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-task1JRg7N/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781955746769051-7083-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.6.234.193:0
--local_ip_for_outbound_sockets=127.6.234.193
--webserver_interface=127.6.234.193
--webserver_port=0
--tserver_master_addrs=127.6.234.254:36839
--keytab_file=/tmp/dist-test-task1JRg7N/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781955746769051-7083-0/krb5kdc/kudu_127.6.234.193.keytab
--principal=kudu/127.6.234.193
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.6.234.212:35613
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--webserver_enabled=0 with env {KRB5CCNAME=/tmp/dist-test-task1JRg7N/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781955746769051-7083-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task1JRg7N/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781955746769051-7083-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task1JRg7N/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781955746769051-7083-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 11:42:46.976504 11226 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 11:42:46.976708 11226 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 11:42:46.976733 11226 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 11:42:46.977950 11226 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 11:42:46.977990 11226 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 11:42:46.978019 11226 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.6.234.193
I20260620 11:42:46.979574 11226 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.6.234.212:35613
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task1JRg7N/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781955746769051-7083-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-task1JRg7N/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781955746769051-7083-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task1JRg7N/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781955746769051-7083-0/krb5kdc/kudu_127.6.234.193.keytab
--principal=kudu/127.6.234.193
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.6.234.193:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--webserver_enabled=false
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task1JRg7N/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781955746769051-7083-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.6.234.193
--webserver_port=0
--tserver_master_addrs=127.6.234.254:36839
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.11226
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.6.234.193
--log_dir=/tmp/dist-test-task1JRg7N/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781955746769051-7083-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 20 Jun 2026 10:32:37 UTC on ccycloud.quasar-bgyofv.root.comops.site
build id 4457173
I20260620 11:42:46.979837 11226 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 11:42:46.980108 11226 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 11:42:46.985095 11234 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 11:42:46.985098 11231 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 11:42:46.985105 11232 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 11:42:46.985394 11226 server_base.cc:1034] running on GCE node
I20260620 11:42:46.985801 11226 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 11:42:46.986066 11226 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 11:42:46.987215 11226 hybrid_clock.cc:648] HybridClock initialized: now 1781955766987200 us; error 29 us; skew 500 ppm
Jun 20 11:42:46 dist-test-slave-cjc6 krb5kdc[11120](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781955766, etypes {rep=17 tkt=17 ses=17}, kudu/127.6.234.193@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 11:42:46.989863 11226 init.cc:377] Logged in from keytab as kudu/127.6.234.193@KRBTEST.COM (short username kudu)
I20260620 11:42:46.990170 11226 fs_manager.cc:359] Metadata directory not provided
I20260620 11:42:46.990216 11226 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 11:42:46.990357 11226 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 11:42:46.991413 11226 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task1JRg7N/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781955746769051-7083-0/minicluster-data/ts-0/data/instance:
uuid: "66d73b1e343f4d55bafe247a62a894da"
format_stamp: "Formatted at 2026-06-20 11:42:46 on dist-test-slave-cjc6"
server_key: "7267b66ae301b2b0263faae9897a7216"
server_key_iv: "82c6e907b6211d35fbad1d2af7d80a8a"
server_key_version: "encryptionkey@0"
I20260620 11:42:46.991767 11226 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task1JRg7N/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781955746769051-7083-0/minicluster-data/ts-0/wal/instance:
uuid: "66d73b1e343f4d55bafe247a62a894da"
format_stamp: "Formatted at 2026-06-20 11:42:46 on dist-test-slave-cjc6"
server_key: "7267b66ae301b2b0263faae9897a7216"
server_key_iv: "82c6e907b6211d35fbad1d2af7d80a8a"
server_key_version: "encryptionkey@0"
I20260620 11:42:46.993108 11226 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.001s
I20260620 11:42:46.993896 11240 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 11:42:46.994086 11226 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.000s
I20260620 11:42:46.994158 11226 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task1JRg7N/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781955746769051-7083-0/minicluster-data/ts-0/data,/tmp/dist-test-task1JRg7N/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781955746769051-7083-0/minicluster-data/ts-0/wal
uuid: "66d73b1e343f4d55bafe247a62a894da"
format_stamp: "Formatted at 2026-06-20 11:42:46 on dist-test-slave-cjc6"
server_key: "7267b66ae301b2b0263faae9897a7216"
server_key_iv: "82c6e907b6211d35fbad1d2af7d80a8a"
server_key_version: "encryptionkey@0"
I20260620 11:42:46.994244 11226 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task1JRg7N/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781955746769051-7083-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-task1JRg7N/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781955746769051-7083-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-task1JRg7N/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781955746769051-7083-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 11:42:47.008121 11226 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 11:42:47.010514 11226 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 11:42:47.010679 11226 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 11:42:47.010924 11226 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 11:42:47.011250 11226 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 11:42:47.011286 11226 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 11:42:47.011307 11226 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 11:42:47.011355 11226 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 11:42:47.017131 11226 rpc_server.cc:307] RPC server started. Bound to: 127.6.234.193:41431
I20260620 11:42:47.017211 11353 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.6.234.193:41431 every 8 connection(s)
I20260620 11:42:47.017477 11226 server_base.cc:1166] Dumped server information to /tmp/dist-test-task1JRg7N/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781955746769051-7083-0/minicluster-data/ts-0/data/info.pb
I20260620 11:42:47.020789 7083 external_mini_cluster.cc:1388] Started /tmp/dist-test-task1JRg7N/build/release/bin/kudu as pid 11226
I20260620 11:42:47.020901 7083 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task1JRg7N/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781955746769051-7083-0/minicluster-data/ts-0/wal/instance
I20260620 11:42:47.021077 7083 external_mini_cluster.cc:1428] Setting key 584d9c40c92b989a0c1580c3a350583c
Jun 20 11:42:47 dist-test-slave-cjc6 krb5kdc[11120](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781955766, etypes {rep=17 tkt=17 ses=17}, kudu/127.6.234.193@KRBTEST.COM for kudu/127.6.234.254@KRBTEST.COM
I20260620 11:42:47.028293 11204 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 11:42:47.018289 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.6.234.193:43741 (local address 127.6.234.254:36839)
0620 11:42:47.018362 (+ 73us) server_negotiation.cc:206] Beginning negotiation
0620 11:42:47.018364 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 11:42:47.018542 (+ 178us) server_negotiation.cc:407] Connection header received
0620 11:42:47.019030 (+ 488us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 11:42:47.019031 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 11:42:47.019044 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 11:42:47.019101 (+ 57us) server_negotiation.cc:226] Negotiated authn=SASL
0620 11:42:47.019990 (+ 889us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 11:42:47.020488 (+ 498us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 11:42:47.021912 (+ 1424us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 11:42:47.022168 (+ 256us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 11:42:47.022233 (+ 65us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 11:42:47.024940 (+ 2707us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 11:42:47.024942 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 11:42:47.024942 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 11:42:47.024948 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0620 11:42:47.026638 (+ 1690us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 11:42:47.027180 (+ 542us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 11:42:47.027181 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 11:42:47.027181 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 11:42:47.027235 (+ 54us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 11:42:47.027571 (+ 336us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 11:42:47.027572 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 11:42:47.027572 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 11:42:47.027713 (+ 141us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 11:42:47.027777 (+ 64us) server_negotiation.cc:1035] Waiting for connection context
0620 11:42:47.028137 (+ 360us) server_negotiation.cc:299] Negotiation successful
0620 11:42:47.028223 (+ 86us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":48}
I20260620 11:42:47.028573 11356 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 11:42:47.018240 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.234.254:36839 (local address 127.6.234.193:43741)
0620 11:42:47.018414 (+ 174us) negotiation.cc:107] Waiting for socket to connect
0620 11:42:47.018454 (+ 40us) client_negotiation.cc:174] Beginning negotiation
0620 11:42:47.018942 (+ 488us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 11:42:47.019155 (+ 213us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 11:42:47.019158 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 11:42:47.019410 (+ 252us) client_negotiation.cc:189] Negotiated authn=SASL
0620 11:42:47.019903 (+ 493us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 11:42:47.019907 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 11:42:47.020569 (+ 662us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 11:42:47.020569 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 11:42:47.021832 (+ 1263us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 11:42:47.021835 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 11:42:47.022256 (+ 421us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 11:42:47.022257 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 11:42:47.022413 (+ 156us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 11:42:47.023087 (+ 674us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 11:42:47.023096 (+ 9us) client_negotiation.cc:656] calling sasl_client_start()
0620 11:42:47.024830 (+ 1734us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 11:42:47.026729 (+ 1899us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 11:42:47.026737 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 11:42:47.026740 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0620 11:42:47.027091 (+ 351us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 11:42:47.027312 (+ 221us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 11:42:47.027313 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 11:42:47.027314 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0620 11:42:47.027468 (+ 154us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 11:42:47.027799 (+ 331us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 11:42:47.027805 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 11:42:47.027989 (+ 184us) client_negotiation.cc:769] Sending connection context
0620 11:42:47.028116 (+ 127us) client_negotiation.cc:240] Negotiation successful
0620 11:42:47.028300 (+ 184us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":129,"thread_start_us":96,"threads_started":1}
I20260620 11:42:47.029177 11354 heartbeater.cc:347] Connected to a master server at 127.6.234.254:36839
I20260620 11:42:47.029263 11354 heartbeater.cc:464] Registering TS with master...
I20260620 11:42:47.029449 11354 heartbeater.cc:511] Master 127.6.234.254:36839 requested a full tablet report, sending...
I20260620 11:42:47.029999 11164 ts_manager.cc:194] Registered new tserver with Master: 66d73b1e343f4d55bafe247a62a894da (127.6.234.193:41431)
WARNING: no policy specified for kudu/127.6.234.194@KRBTEST.COM; defaulting to no policy
I20260620 11:42:47.030612 11164 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.6.234.193@KRBTEST.COM'} at 127.6.234.193:43741
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.6.234.194@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.6.234.194 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task1JRg7N/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781955746769051-7083-0/krb5kdc/kudu_127.6.234.194.keytab.
Entry for principal kudu/127.6.234.194 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task1JRg7N/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781955746769051-7083-0/krb5kdc/kudu_127.6.234.194.keytab.
I20260620 11:42:47.048172 7083 external_mini_cluster.cc:1326] Running /tmp/dist-test-task1JRg7N/build/release/bin/kudu
/tmp/dist-test-task1JRg7N/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task1JRg7N/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781955746769051-7083-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-task1JRg7N/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781955746769051-7083-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task1JRg7N/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781955746769051-7083-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-task1JRg7N/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781955746769051-7083-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.6.234.194:0
--local_ip_for_outbound_sockets=127.6.234.194
--webserver_interface=127.6.234.194
--webserver_port=0
--tserver_master_addrs=127.6.234.254:36839
--keytab_file=/tmp/dist-test-task1JRg7N/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781955746769051-7083-0/krb5kdc/kudu_127.6.234.194.keytab
--principal=kudu/127.6.234.194
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.6.234.212:35613
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--webserver_enabled=0 with env {KRB5CCNAME=/tmp/dist-test-task1JRg7N/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781955746769051-7083-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task1JRg7N/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781955746769051-7083-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task1JRg7N/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781955746769051-7083-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 11:42:47.132390 11359 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 11:42:47.132619 11359 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 11:42:47.132656 11359 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 11:42:47.133822 11359 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 11:42:47.133889 11359 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 11:42:47.133936 11359 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.6.234.194
I20260620 11:42:47.135526 11359 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.6.234.212:35613
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task1JRg7N/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781955746769051-7083-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-task1JRg7N/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781955746769051-7083-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task1JRg7N/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781955746769051-7083-0/krb5kdc/kudu_127.6.234.194.keytab
--principal=kudu/127.6.234.194
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.6.234.194:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--webserver_enabled=false
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task1JRg7N/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781955746769051-7083-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.6.234.194
--webserver_port=0
--tserver_master_addrs=127.6.234.254:36839
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.11359
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.6.234.194
--log_dir=/tmp/dist-test-task1JRg7N/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781955746769051-7083-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 20 Jun 2026 10:32:37 UTC on ccycloud.quasar-bgyofv.root.comops.site
build id 4457173
I20260620 11:42:47.135811 11359 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 11:42:47.136034 11359 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 11:42:47.140856 11364 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 11:42:47.140887 11365 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 11:42:47.140858 11367 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 11:42:47.140992 11359 server_base.cc:1034] running on GCE node
I20260620 11:42:47.141319 11359 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 11:42:47.141558 11359 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 11:42:47.142725 11359 hybrid_clock.cc:648] HybridClock initialized: now 1781955767142708 us; error 47 us; skew 500 ppm
Jun 20 11:42:47 dist-test-slave-cjc6 krb5kdc[11120](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781955767, etypes {rep=17 tkt=17 ses=17}, kudu/127.6.234.194@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 11:42:47.145210 11359 init.cc:377] Logged in from keytab as kudu/127.6.234.194@KRBTEST.COM (short username kudu)
I20260620 11:42:47.145541 11359 fs_manager.cc:359] Metadata directory not provided
I20260620 11:42:47.145615 11359 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 11:42:47.145787 11359 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 11:42:47.146790 11359 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task1JRg7N/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781955746769051-7083-0/minicluster-data/ts-1/data/instance:
uuid: "7c824e5a1c7e4448bbd40d6bf8c5c000"
format_stamp: "Formatted at 2026-06-20 11:42:47 on dist-test-slave-cjc6"
server_key: "f38fb6664d228a6da188501362f7ea4e"
server_key_iv: "bd18274f8f3d9955fd02fbdc74156211"
server_key_version: "encryptionkey@0"
I20260620 11:42:47.147171 11359 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task1JRg7N/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781955746769051-7083-0/minicluster-data/ts-1/wal/instance:
uuid: "7c824e5a1c7e4448bbd40d6bf8c5c000"
format_stamp: "Formatted at 2026-06-20 11:42:47 on dist-test-slave-cjc6"
server_key: "f38fb6664d228a6da188501362f7ea4e"
server_key_iv: "bd18274f8f3d9955fd02fbdc74156211"
server_key_version: "encryptionkey@0"
I20260620 11:42:47.148418 11359 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.002s
I20260620 11:42:47.149176 11373 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 11:42:47.149343 11359 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.001s
I20260620 11:42:47.149441 11359 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task1JRg7N/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781955746769051-7083-0/minicluster-data/ts-1/data,/tmp/dist-test-task1JRg7N/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781955746769051-7083-0/minicluster-data/ts-1/wal
uuid: "7c824e5a1c7e4448bbd40d6bf8c5c000"
format_stamp: "Formatted at 2026-06-20 11:42:47 on dist-test-slave-cjc6"
server_key: "f38fb6664d228a6da188501362f7ea4e"
server_key_iv: "bd18274f8f3d9955fd02fbdc74156211"
server_key_version: "encryptionkey@0"
I20260620 11:42:47.149518 11359 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task1JRg7N/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781955746769051-7083-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-task1JRg7N/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781955746769051-7083-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-task1JRg7N/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781955746769051-7083-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 11:42:47.163074 11359 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 11:42:47.165360 11359 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 11:42:47.165529 11359 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 11:42:47.165836 11359 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 11:42:47.166155 11359 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 11:42:47.166208 11359 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 11:42:47.166244 11359 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 11:42:47.166270 11359 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 11:42:47.171986 11359 rpc_server.cc:307] RPC server started. Bound to: 127.6.234.194:40509
I20260620 11:42:47.172021 11486 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.6.234.194:40509 every 8 connection(s)
I20260620 11:42:47.172284 11359 server_base.cc:1166] Dumped server information to /tmp/dist-test-task1JRg7N/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781955746769051-7083-0/minicluster-data/ts-1/data/info.pb
I20260620 11:42:47.173046 7083 external_mini_cluster.cc:1388] Started /tmp/dist-test-task1JRg7N/build/release/bin/kudu as pid 11359
I20260620 11:42:47.173149 7083 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task1JRg7N/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781955746769051-7083-0/minicluster-data/ts-1/wal/instance
I20260620 11:42:47.173347 7083 external_mini_cluster.cc:1428] Setting key d9a59c4c6708a0478ba27a3948ddc064
Jun 20 11:42:47 dist-test-slave-cjc6 krb5kdc[11120](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781955767, etypes {rep=17 tkt=17 ses=17}, kudu/127.6.234.194@KRBTEST.COM for kudu/127.6.234.254@KRBTEST.COM
WARNING: no policy specified for kudu/127.6.234.195@KRBTEST.COM; defaulting to no policy
I20260620 11:42:47.182395 11204 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 11:42:47.172988 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.6.234.194:45503 (local address 127.6.234.254:36839)
0620 11:42:47.173063 (+ 75us) server_negotiation.cc:206] Beginning negotiation
0620 11:42:47.173065 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 11:42:47.173717 (+ 652us) server_negotiation.cc:407] Connection header received
0620 11:42:47.174238 (+ 521us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 11:42:47.174239 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 11:42:47.174251 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 11:42:47.174308 (+ 57us) server_negotiation.cc:226] Negotiated authn=SASL
0620 11:42:47.175352 (+ 1044us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 11:42:47.175896 (+ 544us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 11:42:47.176948 (+ 1052us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 11:42:47.177127 (+ 179us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 11:42:47.177180 (+ 53us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 11:42:47.179609 (+ 2429us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 11:42:47.179610 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 11:42:47.179611 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 11:42:47.179618 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0620 11:42:47.181001 (+ 1383us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 11:42:47.181420 (+ 419us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 11:42:47.181421 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 11:42:47.181421 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 11:42:47.181480 (+ 59us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 11:42:47.181746 (+ 266us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 11:42:47.181748 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 11:42:47.181748 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 11:42:47.181865 (+ 117us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 11:42:47.181942 (+ 77us) server_negotiation.cc:1035] Waiting for connection context
0620 11:42:47.182213 (+ 271us) server_negotiation.cc:299] Negotiation successful
0620 11:42:47.182304 (+ 91us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":44}
I20260620 11:42:47.182572 11489 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 11:42:47.172933 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.234.254:36839 (local address 127.6.234.194:45503)
0620 11:42:47.173559 (+ 626us) negotiation.cc:107] Waiting for socket to connect
0620 11:42:47.173595 (+ 36us) client_negotiation.cc:174] Beginning negotiation
0620 11:42:47.174122 (+ 527us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 11:42:47.174405 (+ 283us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 11:42:47.174413 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 11:42:47.174671 (+ 258us) client_negotiation.cc:189] Negotiated authn=SASL
0620 11:42:47.175247 (+ 576us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 11:42:47.175253 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 11:42:47.175996 (+ 743us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 11:42:47.175998 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 11:42:47.176867 (+ 869us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 11:42:47.176870 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 11:42:47.177204 (+ 334us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 11:42:47.177206 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 11:42:47.177355 (+ 149us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 11:42:47.177935 (+ 580us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 11:42:47.177945 (+ 10us) client_negotiation.cc:656] calling sasl_client_start()
0620 11:42:47.179513 (+ 1568us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 11:42:47.181097 (+ 1584us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 11:42:47.181104 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 11:42:47.181106 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0620 11:42:47.181328 (+ 222us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 11:42:47.181547 (+ 219us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 11:42:47.181548 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 11:42:47.181549 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0620 11:42:47.181659 (+ 110us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 11:42:47.181932 (+ 273us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 11:42:47.181938 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 11:42:47.182079 (+ 141us) client_negotiation.cc:769] Sending connection context
0620 11:42:47.182182 (+ 103us) client_negotiation.cc:240] Negotiation successful
0620 11:42:47.182337 (+ 155us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":586,"thread_start_us":84,"threads_started":1}
I20260620 11:42:47.183063 11487 heartbeater.cc:347] Connected to a master server at 127.6.234.254:36839
I20260620 11:42:47.183169 11487 heartbeater.cc:464] Registering TS with master...
I20260620 11:42:47.183333 11487 heartbeater.cc:511] Master 127.6.234.254:36839 requested a full tablet report, sending...
I20260620 11:42:47.183717 11164 ts_manager.cc:194] Registered new tserver with Master: 7c824e5a1c7e4448bbd40d6bf8c5c000 (127.6.234.194:40509)
I20260620 11:42:47.184158 11164 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.6.234.194@KRBTEST.COM'} at 127.6.234.194:45503
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.6.234.195@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.6.234.195 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task1JRg7N/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781955746769051-7083-0/krb5kdc/kudu_127.6.234.195.keytab.
Entry for principal kudu/127.6.234.195 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task1JRg7N/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781955746769051-7083-0/krb5kdc/kudu_127.6.234.195.keytab.
I20260620 11:42:47.198707 7083 external_mini_cluster.cc:1326] Running /tmp/dist-test-task1JRg7N/build/release/bin/kudu
/tmp/dist-test-task1JRg7N/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task1JRg7N/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781955746769051-7083-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-task1JRg7N/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781955746769051-7083-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task1JRg7N/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781955746769051-7083-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-task1JRg7N/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781955746769051-7083-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.6.234.195:0
--local_ip_for_outbound_sockets=127.6.234.195
--webserver_interface=127.6.234.195
--webserver_port=0
--tserver_master_addrs=127.6.234.254:36839
--keytab_file=/tmp/dist-test-task1JRg7N/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781955746769051-7083-0/krb5kdc/kudu_127.6.234.195.keytab
--principal=kudu/127.6.234.195
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.6.234.212:35613
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--webserver_enabled=0 with env {KRB5CCNAME=/tmp/dist-test-task1JRg7N/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781955746769051-7083-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task1JRg7N/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781955746769051-7083-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task1JRg7N/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781955746769051-7083-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 11:42:47.277518 11492 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 11:42:47.277791 11492 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 11:42:47.277835 11492 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 11:42:47.279024 11492 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 11:42:47.279096 11492 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 11:42:47.279143 11492 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.6.234.195
I20260620 11:42:47.280623 11492 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.6.234.212:35613
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task1JRg7N/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781955746769051-7083-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-task1JRg7N/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781955746769051-7083-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task1JRg7N/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781955746769051-7083-0/krb5kdc/kudu_127.6.234.195.keytab
--principal=kudu/127.6.234.195
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.6.234.195:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--webserver_enabled=false
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task1JRg7N/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781955746769051-7083-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.6.234.195
--webserver_port=0
--tserver_master_addrs=127.6.234.254:36839
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.11492
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.6.234.195
--log_dir=/tmp/dist-test-task1JRg7N/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781955746769051-7083-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 20 Jun 2026 10:32:37 UTC on ccycloud.quasar-bgyofv.root.comops.site
build id 4457173
I20260620 11:42:47.280884 11492 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 11:42:47.281107 11492 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 11:42:47.286075 11498 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 11:42:47.286100 11497 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 11:42:47.286128 11492 server_base.cc:1034] running on GCE node
W20260620 11:42:47.286099 11500 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 11:42:47.286573 11492 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 11:42:47.286825 11492 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 11:42:47.287984 11492 hybrid_clock.cc:648] HybridClock initialized: now 1781955767287979 us; error 38 us; skew 500 ppm
Jun 20 11:42:47 dist-test-slave-cjc6 krb5kdc[11120](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781955767, etypes {rep=17 tkt=17 ses=17}, kudu/127.6.234.195@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 11:42:47.290571 11492 init.cc:377] Logged in from keytab as kudu/127.6.234.195@KRBTEST.COM (short username kudu)
I20260620 11:42:47.290916 11492 fs_manager.cc:359] Metadata directory not provided
I20260620 11:42:47.290990 11492 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 11:42:47.291142 11492 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 11:42:47.292189 11492 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task1JRg7N/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781955746769051-7083-0/minicluster-data/ts-2/data/instance:
uuid: "e0643dd7e19b4fb5ad0df7418171dd52"
format_stamp: "Formatted at 2026-06-20 11:42:47 on dist-test-slave-cjc6"
server_key: "78d5322f2b53c5c5511db26f5fa619bc"
server_key_iv: "d8ba74f99c7f5a87e44b124b0f715a42"
server_key_version: "encryptionkey@0"
I20260620 11:42:47.292585 11492 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task1JRg7N/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781955746769051-7083-0/minicluster-data/ts-2/wal/instance:
uuid: "e0643dd7e19b4fb5ad0df7418171dd52"
format_stamp: "Formatted at 2026-06-20 11:42:47 on dist-test-slave-cjc6"
server_key: "78d5322f2b53c5c5511db26f5fa619bc"
server_key_iv: "d8ba74f99c7f5a87e44b124b0f715a42"
server_key_version: "encryptionkey@0"
I20260620 11:42:47.293958 11492 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.003s
I20260620 11:42:47.294654 11506 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 11:42:47.294821 11492 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260620 11:42:47.294916 11492 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task1JRg7N/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781955746769051-7083-0/minicluster-data/ts-2/data,/tmp/dist-test-task1JRg7N/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781955746769051-7083-0/minicluster-data/ts-2/wal
uuid: "e0643dd7e19b4fb5ad0df7418171dd52"
format_stamp: "Formatted at 2026-06-20 11:42:47 on dist-test-slave-cjc6"
server_key: "78d5322f2b53c5c5511db26f5fa619bc"
server_key_iv: "d8ba74f99c7f5a87e44b124b0f715a42"
server_key_version: "encryptionkey@0"
I20260620 11:42:47.294996 11492 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task1JRg7N/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781955746769051-7083-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-task1JRg7N/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781955746769051-7083-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-task1JRg7N/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781955746769051-7083-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 11:42:47.304376 11492 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 11:42:47.306118 11492 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 11:42:47.306265 11492 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 11:42:47.306512 11492 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 11:42:47.306803 11492 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 11:42:47.306855 11492 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 11:42:47.306895 11492 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 11:42:47.306923 11492 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 11:42:47.312743 11492 rpc_server.cc:307] RPC server started. Bound to: 127.6.234.195:37491
I20260620 11:42:47.312786 11619 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.6.234.195:37491 every 8 connection(s)
I20260620 11:42:47.313037 11492 server_base.cc:1166] Dumped server information to /tmp/dist-test-task1JRg7N/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781955746769051-7083-0/minicluster-data/ts-2/data/info.pb
Jun 20 11:42:47 dist-test-slave-cjc6 krb5kdc[11120](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781955767, etypes {rep=17 tkt=17 ses=17}, kudu/127.6.234.195@KRBTEST.COM for kudu/127.6.234.254@KRBTEST.COM
I20260620 11:42:47.322099 11204 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 11:42:47.313745 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.6.234.195:37529 (local address 127.6.234.254:36839)
0620 11:42:47.313802 (+ 57us) server_negotiation.cc:206] Beginning negotiation
0620 11:42:47.313807 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 11:42:47.313996 (+ 189us) server_negotiation.cc:407] Connection header received
0620 11:42:47.314496 (+ 500us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 11:42:47.314497 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 11:42:47.314510 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 11:42:47.314566 (+ 56us) server_negotiation.cc:226] Negotiated authn=SASL
0620 11:42:47.315464 (+ 898us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 11:42:47.315966 (+ 502us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 11:42:47.316957 (+ 991us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 11:42:47.317160 (+ 203us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 11:42:47.317244 (+ 84us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 11:42:47.319343 (+ 2099us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 11:42:47.319344 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 11:42:47.319345 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 11:42:47.319352 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0620 11:42:47.320605 (+ 1253us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 11:42:47.321064 (+ 459us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 11:42:47.321066 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 11:42:47.321066 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 11:42:47.321129 (+ 63us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 11:42:47.321390 (+ 261us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 11:42:47.321391 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 11:42:47.321392 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 11:42:47.321512 (+ 120us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 11:42:47.321584 (+ 72us) server_negotiation.cc:1035] Waiting for connection context
0620 11:42:47.321923 (+ 339us) server_negotiation.cc:299] Negotiation successful
0620 11:42:47.322008 (+ 85us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":23}
I20260620 11:42:47.322320 11622 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 11:42:47.313677 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.234.254:36839 (local address 127.6.234.195:37529)
0620 11:42:47.313874 (+ 197us) negotiation.cc:107] Waiting for socket to connect
0620 11:42:47.313915 (+ 41us) client_negotiation.cc:174] Beginning negotiation
0620 11:42:47.314393 (+ 478us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 11:42:47.314630 (+ 237us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 11:42:47.314634 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 11:42:47.314891 (+ 257us) client_negotiation.cc:189] Negotiated authn=SASL
0620 11:42:47.315357 (+ 466us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 11:42:47.315362 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 11:42:47.316056 (+ 694us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 11:42:47.316057 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 11:42:47.316854 (+ 797us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 11:42:47.316856 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 11:42:47.317254 (+ 398us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 11:42:47.317255 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 11:42:47.317380 (+ 125us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 11:42:47.317911 (+ 531us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 11:42:47.317918 (+ 7us) client_negotiation.cc:656] calling sasl_client_start()
0620 11:42:47.319245 (+ 1327us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 11:42:47.320725 (+ 1480us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 11:42:47.320732 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 11:42:47.320734 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0620 11:42:47.320963 (+ 229us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 11:42:47.321199 (+ 236us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 11:42:47.321200 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 11:42:47.321200 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0620 11:42:47.321313 (+ 113us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 11:42:47.321589 (+ 276us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 11:42:47.321595 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 11:42:47.321779 (+ 184us) client_negotiation.cc:769] Sending connection context
0620 11:42:47.321900 (+ 121us) client_negotiation.cc:240] Negotiation successful
0620 11:42:47.322049 (+ 149us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":158,"thread_start_us":101,"threads_started":1}
I20260620 11:42:47.322914 11620 heartbeater.cc:347] Connected to a master server at 127.6.234.254:36839
I20260620 11:42:47.323036 11620 heartbeater.cc:464] Registering TS with master...
I20260620 11:42:47.323096 7083 external_mini_cluster.cc:1388] Started /tmp/dist-test-task1JRg7N/build/release/bin/kudu as pid 11492
I20260620 11:42:47.323208 7083 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task1JRg7N/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781955746769051-7083-0/minicluster-data/ts-2/wal/instance
I20260620 11:42:47.323225 11620 heartbeater.cc:511] Master 127.6.234.254:36839 requested a full tablet report, sending...
I20260620 11:42:47.323433 7083 external_mini_cluster.cc:1428] Setting key 52ff18050179efef7b379845758c3396
I20260620 11:42:47.323616 11164 ts_manager.cc:194] Registered new tserver with Master: e0643dd7e19b4fb5ad0df7418171dd52 (127.6.234.195:37491)
I20260620 11:42:47.324148 11164 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.6.234.195@KRBTEST.COM'} at 127.6.234.195:37529
I20260620 11:42:47.324323 7083 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260620 11:42:47.331037 11204 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 11:42:47.325674 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60664 (local address 127.6.234.254:36839)
0620 11:42:47.325725 (+ 51us) server_negotiation.cc:206] Beginning negotiation
0620 11:42:47.325727 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 11:42:47.325827 (+ 100us) server_negotiation.cc:407] Connection header received
0620 11:42:47.325890 (+ 63us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 11:42:47.325890 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 11:42:47.325898 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 11:42:47.325936 (+ 38us) server_negotiation.cc:226] Negotiated authn=SASL
0620 11:42:47.326448 (+ 512us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 11:42:47.326891 (+ 443us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 11:42:47.327558 (+ 667us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 11:42:47.327730 (+ 172us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 11:42:47.327782 (+ 52us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 11:42:47.328811 (+ 1029us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 11:42:47.328812 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 11:42:47.328813 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 11:42:47.328819 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0620 11:42:47.330171 (+ 1352us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 11:42:47.330462 (+ 291us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 11:42:47.330463 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 11:42:47.330463 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 11:42:47.330506 (+ 43us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 11:42:47.330646 (+ 140us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 11:42:47.330647 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 11:42:47.330647 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 11:42:47.330765 (+ 118us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 11:42:47.330835 (+ 70us) server_negotiation.cc:1035] Waiting for connection context
0620 11:42:47.330924 (+ 89us) server_negotiation.cc:299] Negotiation successful
0620 11:42:47.330977 (+ 53us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":32}
I20260620 11:42:47.332576 11164 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:60664:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260620 11:42:47.334192 11164 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260620 11:42:47.340392 11628 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 11:42:47.336361 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.234.194:40509 (local address 127.0.0.1:57072)
0620 11:42:47.336491 (+ 130us) negotiation.cc:107] Waiting for socket to connect
0620 11:42:47.336504 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0620 11:42:47.336566 (+ 62us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 11:42:47.336798 (+ 232us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 11:42:47.336801 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 11:42:47.336803 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 11:42:47.336869 (+ 66us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 11:42:47.336871 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 11:42:47.338102 (+ 1231us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 11:42:47.338103 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 11:42:47.339307 (+ 1204us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 11:42:47.339309 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 11:42:47.340095 (+ 786us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 11:42:47.340096 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 11:42:47.340185 (+ 89us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 11:42:47.340196 (+ 11us) client_negotiation.cc:769] Sending connection context
0620 11:42:47.340259 (+ 63us) client_negotiation.cc:240] Negotiation successful
0620 11:42:47.340306 (+ 47us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":116,"thread_start_us":57,"threads_started":1}
I20260620 11:42:47.340636 11630 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 11:42:47.336449 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57072 (local address 127.6.234.194:40509)
0620 11:42:47.336649 (+ 200us) server_negotiation.cc:206] Beginning negotiation
0620 11:42:47.336653 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 11:42:47.336663 (+ 10us) server_negotiation.cc:407] Connection header received
0620 11:42:47.336702 (+ 39us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 11:42:47.336705 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 11:42:47.336720 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 11:42:47.336794 (+ 74us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 11:42:47.336929 (+ 135us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 11:42:47.337999 (+ 1070us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 11:42:47.339407 (+ 1408us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 11:42:47.339936 (+ 529us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 11:42:47.340004 (+ 68us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 11:42:47.340059 (+ 55us) server_negotiation.cc:1035] Waiting for connection context
0620 11:42:47.340501 (+ 442us) server_negotiation.cc:299] Negotiation successful
0620 11:42:47.340543 (+ 42us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":173,"thread_start_us":98,"threads_started":1}
I20260620 11:42:47.340978 11631 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 11:42:47.336674 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.234.195:37491 (local address 127.0.0.1:48164)
0620 11:42:47.336991 (+ 317us) negotiation.cc:107] Waiting for socket to connect
0620 11:42:47.337002 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0620 11:42:47.337075 (+ 73us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 11:42:47.337426 (+ 351us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 11:42:47.337427 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 11:42:47.337429 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 11:42:47.337467 (+ 38us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 11:42:47.337469 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 11:42:47.338558 (+ 1089us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 11:42:47.338559 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 11:42:47.339772 (+ 1213us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 11:42:47.339775 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 11:42:47.340725 (+ 950us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 11:42:47.340726 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 11:42:47.340791 (+ 65us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 11:42:47.340795 (+ 4us) client_negotiation.cc:769] Sending connection context
0620 11:42:47.340873 (+ 78us) client_negotiation.cc:240] Negotiation successful
0620 11:42:47.340911 (+ 38us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":304,"mutex_wait_us":63,"thread_start_us":52,"threads_started":1}
I20260620 11:42:47.341084 11629 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 11:42:47.336340 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.234.193:41431 (local address 127.0.0.1:45182)
0620 11:42:47.336464 (+ 124us) negotiation.cc:107] Waiting for socket to connect
0620 11:42:47.336502 (+ 38us) client_negotiation.cc:174] Beginning negotiation
0620 11:42:47.336635 (+ 133us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 11:42:47.337148 (+ 513us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 11:42:47.337149 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 11:42:47.337151 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 11:42:47.337203 (+ 52us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 11:42:47.337205 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 11:42:47.338601 (+ 1396us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 11:42:47.338603 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 11:42:47.339908 (+ 1305us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 11:42:47.339909 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 11:42:47.340616 (+ 707us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 11:42:47.340617 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 11:42:47.340681 (+ 64us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 11:42:47.340685 (+ 4us) client_negotiation.cc:769] Sending connection context
0620 11:42:47.340939 (+ 254us) client_negotiation.cc:240] Negotiation successful
0620 11:42:47.340983 (+ 44us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":98,"mutex_wait_us":40,"thread_start_us":55,"threads_started":1}
I20260620 11:42:47.341300 11632 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 11:42:47.336758 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48164 (local address 127.6.234.195:37491)
0620 11:42:47.337187 (+ 429us) server_negotiation.cc:206] Beginning negotiation
0620 11:42:47.337191 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 11:42:47.337199 (+ 8us) server_negotiation.cc:407] Connection header received
0620 11:42:47.337234 (+ 35us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 11:42:47.337236 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 11:42:47.337251 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 11:42:47.337345 (+ 94us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 11:42:47.337553 (+ 208us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 11:42:47.338457 (+ 904us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 11:42:47.339865 (+ 1408us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 11:42:47.340572 (+ 707us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 11:42:47.340639 (+ 67us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 11:42:47.340689 (+ 50us) server_negotiation.cc:1035] Waiting for connection context
0620 11:42:47.341184 (+ 495us) server_negotiation.cc:299] Negotiation successful
0620 11:42:47.341225 (+ 41us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":396,"thread_start_us":75,"threads_started":1}
I20260620 11:42:47.341445 11633 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 11:42:47.336818 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45182 (local address 127.6.234.193:41431)
0620 11:42:47.336970 (+ 152us) server_negotiation.cc:206] Beginning negotiation
0620 11:42:47.336975 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 11:42:47.336985 (+ 10us) server_negotiation.cc:407] Connection header received
0620 11:42:47.337028 (+ 43us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 11:42:47.337030 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 11:42:47.337046 (+ 16us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 11:42:47.337478 (+ 432us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 11:42:47.337564 (+ 86us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 11:42:47.338479 (+ 915us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 11:42:47.339992 (+ 1513us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 11:42:47.340510 (+ 518us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 11:42:47.341228 (+ 718us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 11:42:47.341296 (+ 68us) server_negotiation.cc:1035] Waiting for connection context
0620 11:42:47.341323 (+ 27us) server_negotiation.cc:299] Negotiation successful
0620 11:42:47.341371 (+ 48us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":119,"thread_start_us":91,"threads_started":1}
I20260620 11:42:47.341332 11421 tablet_service.cc:1467] Processing CreateTablet for tablet 86337c23f495454f8649f5e1016d8b5f (DEFAULT_TABLE table=test-table [id=358e02c07bb34f619b13493f70ba6514]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 11:42:47.341761 11421 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 86337c23f495454f8649f5e1016d8b5f. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 11:42:47.342113 11288 tablet_service.cc:1467] Processing CreateTablet for tablet 86337c23f495454f8649f5e1016d8b5f (DEFAULT_TABLE table=test-table [id=358e02c07bb34f619b13493f70ba6514]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 11:42:47.342110 11554 tablet_service.cc:1467] Processing CreateTablet for tablet 86337c23f495454f8649f5e1016d8b5f (DEFAULT_TABLE table=test-table [id=358e02c07bb34f619b13493f70ba6514]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 11:42:47.342429 11554 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 86337c23f495454f8649f5e1016d8b5f. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 11:42:47.342429 11288 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 86337c23f495454f8649f5e1016d8b5f. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 11:42:47.344343 11634 tablet_bootstrap.cc:492] T 86337c23f495454f8649f5e1016d8b5f P 7c824e5a1c7e4448bbd40d6bf8c5c000: Bootstrap starting.
I20260620 11:42:47.345014 11634 tablet_bootstrap.cc:654] T 86337c23f495454f8649f5e1016d8b5f P 7c824e5a1c7e4448bbd40d6bf8c5c000: Neither blocks nor log segments found. Creating new log.
I20260620 11:42:47.345592 11636 tablet_bootstrap.cc:492] T 86337c23f495454f8649f5e1016d8b5f P e0643dd7e19b4fb5ad0df7418171dd52: Bootstrap starting.
I20260620 11:42:47.345587 11634 log.cc:824] T 86337c23f495454f8649f5e1016d8b5f P 7c824e5a1c7e4448bbd40d6bf8c5c000: Log is configured to *not* fsync() on all Append() calls
I20260620 11:42:47.346239 11636 tablet_bootstrap.cc:654] T 86337c23f495454f8649f5e1016d8b5f P e0643dd7e19b4fb5ad0df7418171dd52: Neither blocks nor log segments found. Creating new log.
I20260620 11:42:47.346392 11634 tablet_bootstrap.cc:492] T 86337c23f495454f8649f5e1016d8b5f P 7c824e5a1c7e4448bbd40d6bf8c5c000: No bootstrap required, opened a new log
I20260620 11:42:47.346468 11634 ts_tablet_manager.cc:1397] T 86337c23f495454f8649f5e1016d8b5f P 7c824e5a1c7e4448bbd40d6bf8c5c000: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20260620 11:42:47.346832 11636 log.cc:824] T 86337c23f495454f8649f5e1016d8b5f P e0643dd7e19b4fb5ad0df7418171dd52: Log is configured to *not* fsync() on all Append() calls
I20260620 11:42:47.347378 11635 tablet_bootstrap.cc:492] T 86337c23f495454f8649f5e1016d8b5f P 66d73b1e343f4d55bafe247a62a894da: Bootstrap starting.
I20260620 11:42:47.347662 11636 tablet_bootstrap.cc:492] T 86337c23f495454f8649f5e1016d8b5f P e0643dd7e19b4fb5ad0df7418171dd52: No bootstrap required, opened a new log
I20260620 11:42:47.347747 11636 ts_tablet_manager.cc:1397] T 86337c23f495454f8649f5e1016d8b5f P e0643dd7e19b4fb5ad0df7418171dd52: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20260620 11:42:47.347988 11635 tablet_bootstrap.cc:654] T 86337c23f495454f8649f5e1016d8b5f P 66d73b1e343f4d55bafe247a62a894da: Neither blocks nor log segments found. Creating new log.
I20260620 11:42:47.348582 11635 log.cc:824] T 86337c23f495454f8649f5e1016d8b5f P 66d73b1e343f4d55bafe247a62a894da: Log is configured to *not* fsync() on all Append() calls
I20260620 11:42:47.348745 11634 raft_consensus.cc:348] T 86337c23f495454f8649f5e1016d8b5f P 7c824e5a1c7e4448bbd40d6bf8c5c000 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "66d73b1e343f4d55bafe247a62a894da" member_type: VOTER last_known_addr { host: "127.6.234.193" port: 41431 } } peers { permanent_uuid: "7c824e5a1c7e4448bbd40d6bf8c5c000" member_type: VOTER last_known_addr { host: "127.6.234.194" port: 40509 } } peers { permanent_uuid: "e0643dd7e19b4fb5ad0df7418171dd52" member_type: VOTER last_known_addr { host: "127.6.234.195" port: 37491 } }
I20260620 11:42:47.348937 11634 raft_consensus.cc:374] T 86337c23f495454f8649f5e1016d8b5f P 7c824e5a1c7e4448bbd40d6bf8c5c000 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 11:42:47.348984 11634 raft_consensus.cc:729] T 86337c23f495454f8649f5e1016d8b5f P 7c824e5a1c7e4448bbd40d6bf8c5c000 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7c824e5a1c7e4448bbd40d6bf8c5c000, State: Initialized, Role: FOLLOWER
I20260620 11:42:47.349094 11634 consensus_queue.cc:260] T 86337c23f495454f8649f5e1016d8b5f P 7c824e5a1c7e4448bbd40d6bf8c5c000 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "66d73b1e343f4d55bafe247a62a894da" member_type: VOTER last_known_addr { host: "127.6.234.193" port: 41431 } } peers { permanent_uuid: "7c824e5a1c7e4448bbd40d6bf8c5c000" member_type: VOTER last_known_addr { host: "127.6.234.194" port: 40509 } } peers { permanent_uuid: "e0643dd7e19b4fb5ad0df7418171dd52" member_type: VOTER last_known_addr { host: "127.6.234.195" port: 37491 } }
I20260620 11:42:47.349377 11635 tablet_bootstrap.cc:492] T 86337c23f495454f8649f5e1016d8b5f P 66d73b1e343f4d55bafe247a62a894da: No bootstrap required, opened a new log
I20260620 11:42:47.349388 11634 ts_tablet_manager.cc:1428] T 86337c23f495454f8649f5e1016d8b5f P 7c824e5a1c7e4448bbd40d6bf8c5c000: Time spent starting tablet: real 0.003s user 0.003s sys 0.000s
I20260620 11:42:47.349447 11635 ts_tablet_manager.cc:1397] T 86337c23f495454f8649f5e1016d8b5f P 66d73b1e343f4d55bafe247a62a894da: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20260620 11:42:47.349502 11487 heartbeater.cc:503] Master 127.6.234.254:36839 was elected leader, sending a full tablet report...
I20260620 11:42:47.349453 11636 raft_consensus.cc:348] T 86337c23f495454f8649f5e1016d8b5f P e0643dd7e19b4fb5ad0df7418171dd52 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "66d73b1e343f4d55bafe247a62a894da" member_type: VOTER last_known_addr { host: "127.6.234.193" port: 41431 } } peers { permanent_uuid: "7c824e5a1c7e4448bbd40d6bf8c5c000" member_type: VOTER last_known_addr { host: "127.6.234.194" port: 40509 } } peers { permanent_uuid: "e0643dd7e19b4fb5ad0df7418171dd52" member_type: VOTER last_known_addr { host: "127.6.234.195" port: 37491 } }
I20260620 11:42:47.349610 11636 raft_consensus.cc:374] T 86337c23f495454f8649f5e1016d8b5f P e0643dd7e19b4fb5ad0df7418171dd52 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 11:42:47.349649 11636 raft_consensus.cc:729] T 86337c23f495454f8649f5e1016d8b5f P e0643dd7e19b4fb5ad0df7418171dd52 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e0643dd7e19b4fb5ad0df7418171dd52, State: Initialized, Role: FOLLOWER
I20260620 11:42:47.349794 11636 consensus_queue.cc:260] T 86337c23f495454f8649f5e1016d8b5f P e0643dd7e19b4fb5ad0df7418171dd52 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "66d73b1e343f4d55bafe247a62a894da" member_type: VOTER last_known_addr { host: "127.6.234.193" port: 41431 } } peers { permanent_uuid: "7c824e5a1c7e4448bbd40d6bf8c5c000" member_type: VOTER last_known_addr { host: "127.6.234.194" port: 40509 } } peers { permanent_uuid: "e0643dd7e19b4fb5ad0df7418171dd52" member_type: VOTER last_known_addr { host: "127.6.234.195" port: 37491 } }
I20260620 11:42:47.350123 11636 ts_tablet_manager.cc:1428] T 86337c23f495454f8649f5e1016d8b5f P e0643dd7e19b4fb5ad0df7418171dd52: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20260620 11:42:47.350144 11620 heartbeater.cc:503] Master 127.6.234.254:36839 was elected leader, sending a full tablet report...
I20260620 11:42:47.351497 11635 raft_consensus.cc:348] T 86337c23f495454f8649f5e1016d8b5f P 66d73b1e343f4d55bafe247a62a894da [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "66d73b1e343f4d55bafe247a62a894da" member_type: VOTER last_known_addr { host: "127.6.234.193" port: 41431 } } peers { permanent_uuid: "7c824e5a1c7e4448bbd40d6bf8c5c000" member_type: VOTER last_known_addr { host: "127.6.234.194" port: 40509 } } peers { permanent_uuid: "e0643dd7e19b4fb5ad0df7418171dd52" member_type: VOTER last_known_addr { host: "127.6.234.195" port: 37491 } }
I20260620 11:42:47.351611 11635 raft_consensus.cc:374] T 86337c23f495454f8649f5e1016d8b5f P 66d73b1e343f4d55bafe247a62a894da [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 11:42:47.351631 11635 raft_consensus.cc:729] T 86337c23f495454f8649f5e1016d8b5f P 66d73b1e343f4d55bafe247a62a894da [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 66d73b1e343f4d55bafe247a62a894da, State: Initialized, Role: FOLLOWER
I20260620 11:42:47.351702 11635 consensus_queue.cc:260] T 86337c23f495454f8649f5e1016d8b5f P 66d73b1e343f4d55bafe247a62a894da [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "66d73b1e343f4d55bafe247a62a894da" member_type: VOTER last_known_addr { host: "127.6.234.193" port: 41431 } } peers { permanent_uuid: "7c824e5a1c7e4448bbd40d6bf8c5c000" member_type: VOTER last_known_addr { host: "127.6.234.194" port: 40509 } } peers { permanent_uuid: "e0643dd7e19b4fb5ad0df7418171dd52" member_type: VOTER last_known_addr { host: "127.6.234.195" port: 37491 } }
I20260620 11:42:47.351938 11635 ts_tablet_manager.cc:1428] T 86337c23f495454f8649f5e1016d8b5f P 66d73b1e343f4d55bafe247a62a894da: Time spent starting tablet: real 0.002s user 0.003s sys 0.000s
I20260620 11:42:47.351939 11354 heartbeater.cc:503] Master 127.6.234.254:36839 was elected leader, sending a full tablet report...
I20260620 11:42:47.415261 11641 raft_consensus.cc:482] T 86337c23f495454f8649f5e1016d8b5f P e0643dd7e19b4fb5ad0df7418171dd52 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260620 11:42:47.415381 11641 raft_consensus.cc:504] T 86337c23f495454f8649f5e1016d8b5f P e0643dd7e19b4fb5ad0df7418171dd52 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "66d73b1e343f4d55bafe247a62a894da" member_type: VOTER last_known_addr { host: "127.6.234.193" port: 41431 } } peers { permanent_uuid: "7c824e5a1c7e4448bbd40d6bf8c5c000" member_type: VOTER last_known_addr { host: "127.6.234.194" port: 40509 } } peers { permanent_uuid: "e0643dd7e19b4fb5ad0df7418171dd52" member_type: VOTER last_known_addr { host: "127.6.234.195" port: 37491 } }
I20260620 11:42:47.415830 11641 leader_election.cc:288] T 86337c23f495454f8649f5e1016d8b5f P e0643dd7e19b4fb5ad0df7418171dd52 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 66d73b1e343f4d55bafe247a62a894da (127.6.234.193:41431), 7c824e5a1c7e4448bbd40d6bf8c5c000 (127.6.234.194:40509)
I20260620 11:42:47.418751 11633 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 11:42:47.416064 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.6.234.195:58863 (local address 127.6.234.193:41431)
0620 11:42:47.416106 (+ 42us) server_negotiation.cc:206] Beginning negotiation
0620 11:42:47.416108 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 11:42:47.416117 (+ 9us) server_negotiation.cc:407] Connection header received
0620 11:42:47.416145 (+ 28us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 11:42:47.416146 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 11:42:47.416156 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 11:42:47.416220 (+ 64us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 11:42:47.416427 (+ 207us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 11:42:47.416963 (+ 536us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 11:42:47.418047 (+ 1084us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 11:42:47.418386 (+ 339us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 11:42:47.418442 (+ 56us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 11:42:47.418471 (+ 29us) server_negotiation.cc:1035] Waiting for connection context
0620 11:42:47.418627 (+ 156us) server_negotiation.cc:299] Negotiation successful
0620 11:42:47.418667 (+ 40us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":22}
I20260620 11:42:47.418751 11622 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 11:42:47.415980 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.234.193:41431 (local address 127.6.234.195:58863)
0620 11:42:47.416035 (+ 55us) negotiation.cc:107] Waiting for socket to connect
0620 11:42:47.416046 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0620 11:42:47.416090 (+ 44us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 11:42:47.416265 (+ 175us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 11:42:47.416266 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 11:42:47.416269 (+ 3us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 11:42:47.416349 (+ 80us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 11:42:47.416351 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 11:42:47.417047 (+ 696us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 11:42:47.417048 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 11:42:47.417933 (+ 885us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 11:42:47.417935 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 11:42:47.418464 (+ 529us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 11:42:47.418465 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 11:42:47.418552 (+ 87us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 11:42:47.418555 (+ 3us) client_negotiation.cc:769] Sending connection context
0620 11:42:47.418598 (+ 43us) client_negotiation.cc:240] Negotiation successful
0620 11:42:47.418648 (+ 50us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":35,"mutex_wait_us":32}
I20260620 11:42:47.418785 11643 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 11:42:47.416007 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.234.194:40509 (local address 127.6.234.195:43093)
0620 11:42:47.416190 (+ 183us) negotiation.cc:107] Waiting for socket to connect
0620 11:42:47.416199 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0620 11:42:47.416247 (+ 48us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 11:42:47.416423 (+ 176us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 11:42:47.416424 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 11:42:47.416426 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 11:42:47.416494 (+ 68us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 11:42:47.416495 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 11:42:47.417117 (+ 622us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 11:42:47.417118 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 11:42:47.417902 (+ 784us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 11:42:47.417904 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 11:42:47.418500 (+ 596us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 11:42:47.418501 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 11:42:47.418569 (+ 68us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 11:42:47.418573 (+ 4us) client_negotiation.cc:769] Sending connection context
0620 11:42:47.418648 (+ 75us) client_negotiation.cc:240] Negotiation successful
0620 11:42:47.418683 (+ 35us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":170,"mutex_wait_us":2,"thread_start_us":109,"threads_started":1}
I20260620 11:42:47.418787 11630 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 11:42:47.416112 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.6.234.195:43093 (local address 127.6.234.194:40509)
0620 11:42:47.416159 (+ 47us) server_negotiation.cc:206] Beginning negotiation
0620 11:42:47.416160 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0620 11:42:47.416287 (+ 127us) server_negotiation.cc:407] Connection header received
0620 11:42:47.416315 (+ 28us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 11:42:47.416316 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 11:42:47.416328 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 11:42:47.416391 (+ 63us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 11:42:47.416565 (+ 174us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 11:42:47.417039 (+ 474us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 11:42:47.417992 (+ 953us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 11:42:47.418395 (+ 403us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 11:42:47.418486 (+ 91us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 11:42:47.418494 (+ 8us) server_negotiation.cc:1035] Waiting for connection context
0620 11:42:47.418670 (+ 176us) server_negotiation.cc:299] Negotiation successful
0620 11:42:47.418709 (+ 39us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":22}
I20260620 11:42:47.419112 11441 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "86337c23f495454f8649f5e1016d8b5f" candidate_uuid: "e0643dd7e19b4fb5ad0df7418171dd52" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7c824e5a1c7e4448bbd40d6bf8c5c000" is_pre_election: true
I20260620 11:42:47.419095 11308 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "86337c23f495454f8649f5e1016d8b5f" candidate_uuid: "e0643dd7e19b4fb5ad0df7418171dd52" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "66d73b1e343f4d55bafe247a62a894da" is_pre_election: true
I20260620 11:42:47.419260 11441 raft_consensus.cc:2454] T 86337c23f495454f8649f5e1016d8b5f P 7c824e5a1c7e4448bbd40d6bf8c5c000 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e0643dd7e19b4fb5ad0df7418171dd52 in term 0.
I20260620 11:42:47.419260 11308 raft_consensus.cc:2454] T 86337c23f495454f8649f5e1016d8b5f P 66d73b1e343f4d55bafe247a62a894da [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e0643dd7e19b4fb5ad0df7418171dd52 in term 0.
I20260620 11:42:47.419502 11508 leader_election.cc:302] T 86337c23f495454f8649f5e1016d8b5f P e0643dd7e19b4fb5ad0df7418171dd52 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 7c824e5a1c7e4448bbd40d6bf8c5c000, e0643dd7e19b4fb5ad0df7418171dd52; no voters:
I20260620 11:42:47.419628 11641 raft_consensus.cc:2781] T 86337c23f495454f8649f5e1016d8b5f P e0643dd7e19b4fb5ad0df7418171dd52 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260620 11:42:47.419683 11641 raft_consensus.cc:482] T 86337c23f495454f8649f5e1016d8b5f P e0643dd7e19b4fb5ad0df7418171dd52 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260620 11:42:47.419708 11641 raft_consensus.cc:3037] T 86337c23f495454f8649f5e1016d8b5f P e0643dd7e19b4fb5ad0df7418171dd52 [term 0 FOLLOWER]: Advancing to term 1
I20260620 11:42:47.420373 11641 raft_consensus.cc:504] T 86337c23f495454f8649f5e1016d8b5f P e0643dd7e19b4fb5ad0df7418171dd52 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "66d73b1e343f4d55bafe247a62a894da" member_type: VOTER last_known_addr { host: "127.6.234.193" port: 41431 } } peers { permanent_uuid: "7c824e5a1c7e4448bbd40d6bf8c5c000" member_type: VOTER last_known_addr { host: "127.6.234.194" port: 40509 } } peers { permanent_uuid: "e0643dd7e19b4fb5ad0df7418171dd52" member_type: VOTER last_known_addr { host: "127.6.234.195" port: 37491 } }
I20260620 11:42:47.420513 11641 leader_election.cc:288] T 86337c23f495454f8649f5e1016d8b5f P e0643dd7e19b4fb5ad0df7418171dd52 [CANDIDATE]: Term 1 election: Requested vote from peers 66d73b1e343f4d55bafe247a62a894da (127.6.234.193:41431), 7c824e5a1c7e4448bbd40d6bf8c5c000 (127.6.234.194:40509)
I20260620 11:42:47.420668 11441 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "86337c23f495454f8649f5e1016d8b5f" candidate_uuid: "e0643dd7e19b4fb5ad0df7418171dd52" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7c824e5a1c7e4448bbd40d6bf8c5c000"
I20260620 11:42:47.420658 11308 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "86337c23f495454f8649f5e1016d8b5f" candidate_uuid: "e0643dd7e19b4fb5ad0df7418171dd52" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "66d73b1e343f4d55bafe247a62a894da"
I20260620 11:42:47.420745 11308 raft_consensus.cc:3037] T 86337c23f495454f8649f5e1016d8b5f P 66d73b1e343f4d55bafe247a62a894da [term 0 FOLLOWER]: Advancing to term 1
I20260620 11:42:47.420745 11441 raft_consensus.cc:3037] T 86337c23f495454f8649f5e1016d8b5f P 7c824e5a1c7e4448bbd40d6bf8c5c000 [term 0 FOLLOWER]: Advancing to term 1
I20260620 11:42:47.421653 11308 raft_consensus.cc:2454] T 86337c23f495454f8649f5e1016d8b5f P 66d73b1e343f4d55bafe247a62a894da [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e0643dd7e19b4fb5ad0df7418171dd52 in term 1.
I20260620 11:42:47.421653 11441 raft_consensus.cc:2454] T 86337c23f495454f8649f5e1016d8b5f P 7c824e5a1c7e4448bbd40d6bf8c5c000 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e0643dd7e19b4fb5ad0df7418171dd52 in term 1.
I20260620 11:42:47.421826 11508 leader_election.cc:302] T 86337c23f495454f8649f5e1016d8b5f P e0643dd7e19b4fb5ad0df7418171dd52 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 3 yes votes; 0 no votes. yes voters: 66d73b1e343f4d55bafe247a62a894da, 7c824e5a1c7e4448bbd40d6bf8c5c000, e0643dd7e19b4fb5ad0df7418171dd52; no voters:
I20260620 11:42:47.421939 11641 raft_consensus.cc:2781] T 86337c23f495454f8649f5e1016d8b5f P e0643dd7e19b4fb5ad0df7418171dd52 [term 1 FOLLOWER]: Leader election won for term 1
I20260620 11:42:47.422118 11641 raft_consensus.cc:686] T 86337c23f495454f8649f5e1016d8b5f P e0643dd7e19b4fb5ad0df7418171dd52 [term 1 LEADER]: Becoming Leader. State: Replica: e0643dd7e19b4fb5ad0df7418171dd52, State: Running, Role: LEADER
I20260620 11:42:47.422230 11641 consensus_queue.cc:237] T 86337c23f495454f8649f5e1016d8b5f P e0643dd7e19b4fb5ad0df7418171dd52 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "66d73b1e343f4d55bafe247a62a894da" member_type: VOTER last_known_addr { host: "127.6.234.193" port: 41431 } } peers { permanent_uuid: "7c824e5a1c7e4448bbd40d6bf8c5c000" member_type: VOTER last_known_addr { host: "127.6.234.194" port: 40509 } } peers { permanent_uuid: "e0643dd7e19b4fb5ad0df7418171dd52" member_type: VOTER last_known_addr { host: "127.6.234.195" port: 37491 } }
W20260620 11:42:47.422742 11488 tablet.cc:2369] T 86337c23f495454f8649f5e1016d8b5f P 7c824e5a1c7e4448bbd40d6bf8c5c000: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260620 11:42:47.423107 11163 catalog_manager.cc:5495] T 86337c23f495454f8649f5e1016d8b5f P e0643dd7e19b4fb5ad0df7418171dd52 reported cstate change: term changed from 0 to 1, leader changed from <none> to e0643dd7e19b4fb5ad0df7418171dd52 (127.6.234.195). New cstate: current_term: 1 leader_uuid: "e0643dd7e19b4fb5ad0df7418171dd52" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "66d73b1e343f4d55bafe247a62a894da" member_type: VOTER last_known_addr { host: "127.6.234.193" port: 41431 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "7c824e5a1c7e4448bbd40d6bf8c5c000" member_type: VOTER last_known_addr { host: "127.6.234.194" port: 40509 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "e0643dd7e19b4fb5ad0df7418171dd52" member_type: VOTER last_known_addr { host: "127.6.234.195" port: 37491 } health_report { overall_health: HEALTHY } } }
I20260620 11:42:47.435094 11632 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 11:42:47.432726 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48180 (local address 127.6.234.195:37491)
0620 11:42:47.432791 (+ 65us) server_negotiation.cc:206] Beginning negotiation
0620 11:42:47.432793 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 11:42:47.432800 (+ 7us) server_negotiation.cc:407] Connection header received
0620 11:42:47.432848 (+ 48us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 11:42:47.432850 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 11:42:47.432865 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 11:42:47.432925 (+ 60us) server_negotiation.cc:226] Negotiated authn=TOKEN
0620 11:42:47.433061 (+ 136us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 11:42:47.433509 (+ 448us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 11:42:47.434357 (+ 848us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 11:42:47.434541 (+ 184us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 11:42:47.434604 (+ 63us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 11:42:47.434763 (+ 159us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0620 11:42:47.434872 (+ 109us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0620 11:42:47.434935 (+ 63us) server_negotiation.cc:1035] Waiting for connection context
0620 11:42:47.435004 (+ 69us) server_negotiation.cc:299] Negotiation successful
0620 11:42:47.435022 (+ 18us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":36}
I20260620 11:42:47.436920 11441 raft_consensus.cc:1261] T 86337c23f495454f8649f5e1016d8b5f P 7c824e5a1c7e4448bbd40d6bf8c5c000 [term 1 FOLLOWER]: Refusing update from remote peer e0643dd7e19b4fb5ad0df7418171dd52: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 11:42:47.436921 11308 raft_consensus.cc:1261] T 86337c23f495454f8649f5e1016d8b5f P 66d73b1e343f4d55bafe247a62a894da [term 1 FOLLOWER]: Refusing update from remote peer e0643dd7e19b4fb5ad0df7418171dd52: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 11:42:47.437309 11644 consensus_queue.cc:1035] T 86337c23f495454f8649f5e1016d8b5f P e0643dd7e19b4fb5ad0df7418171dd52 [LEADER]: Connected to new peer: Peer: permanent_uuid: "7c824e5a1c7e4448bbd40d6bf8c5c000" member_type: VOTER last_known_addr { host: "127.6.234.194" port: 40509 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 11:42:47.437469 11641 consensus_queue.cc:1035] T 86337c23f495454f8649f5e1016d8b5f P e0643dd7e19b4fb5ad0df7418171dd52 [LEADER]: Connected to new peer: Peer: permanent_uuid: "66d73b1e343f4d55bafe247a62a894da" member_type: VOTER last_known_addr { host: "127.6.234.193" port: 41431 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 11:42:47.439716 11647 mvcc.cc:205] Tried to move back new op lower bound from 7298890823419289600 to 7298890823362146304. Current Snapshot: MvccSnapshot[applied={T|T < 7298890823419289600}]
I20260620 11:42:47.440367 11648 mvcc.cc:205] Tried to move back new op lower bound from 7298890823419289600 to 7298890823362146304. Current Snapshot: MvccSnapshot[applied={T|T < 7298890823419289600}]
I20260620 11:42:47.442425 11164 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:60664:
table { table_name: "test-table" } modify_external_catalogs: true
I20260620 11:42:47.442512 11164 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:60664:
table { table_name: "test-table" } modify_external_catalogs: true
I20260620 11:42:47.443377 11164 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 46b14fdaa32d4783bb3a63a5574997e5: Sending DeleteTablet for 3 replicas of tablet 86337c23f495454f8649f5e1016d8b5f
I20260620 11:42:47.443696 11288 tablet_service.cc:1514] Processing DeleteTablet for tablet 86337c23f495454f8649f5e1016d8b5f with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 11:42:47 UTC) from {username='kudu', principal='kudu/127.6.234.254@KRBTEST.COM'} at 127.0.0.1:45182
I20260620 11:42:47.443715 11554 tablet_service.cc:1514] Processing DeleteTablet for tablet 86337c23f495454f8649f5e1016d8b5f with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 11:42:47 UTC) from {username='kudu', principal='kudu/127.6.234.254@KRBTEST.COM'} at 127.0.0.1:48164
I20260620 11:42:47.443851 11421 tablet_service.cc:1514] Processing DeleteTablet for tablet 86337c23f495454f8649f5e1016d8b5f with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 11:42:47 UTC) from {username='kudu', principal='kudu/127.6.234.254@KRBTEST.COM'} at 127.0.0.1:57072
I20260620 11:42:47.444082 11659 tablet_replica.cc:330] T 86337c23f495454f8649f5e1016d8b5f P 7c824e5a1c7e4448bbd40d6bf8c5c000: stopping tablet replica
I20260620 11:42:47.444213 11659 raft_consensus.cc:2229] T 86337c23f495454f8649f5e1016d8b5f P 7c824e5a1c7e4448bbd40d6bf8c5c000 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260620 11:42:47.444365 11659 raft_consensus.cc:2258] T 86337c23f495454f8649f5e1016d8b5f P 7c824e5a1c7e4448bbd40d6bf8c5c000 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 11:42:47.444458 7083 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task1JRg7N/build/release/bin/kudu with pid 11226
I20260620 11:42:47.445109 11658 tablet_replica.cc:330] T 86337c23f495454f8649f5e1016d8b5f P e0643dd7e19b4fb5ad0df7418171dd52: stopping tablet replica
I20260620 11:42:47.445207 11658 raft_consensus.cc:2229] T 86337c23f495454f8649f5e1016d8b5f P e0643dd7e19b4fb5ad0df7418171dd52 [term 1 LEADER]: Raft consensus shutting down.
I20260620 11:42:47.445322 11658 raft_consensus.cc:2258] T 86337c23f495454f8649f5e1016d8b5f P e0643dd7e19b4fb5ad0df7418171dd52 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 11:42:47.445788 11658 ts_tablet_manager.cc:1905] T 86337c23f495454f8649f5e1016d8b5f P e0643dd7e19b4fb5ad0df7418171dd52: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 11:42:47.446218 11659 ts_tablet_manager.cc:1905] T 86337c23f495454f8649f5e1016d8b5f P 7c824e5a1c7e4448bbd40d6bf8c5c000: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 11:42:47.447664 11658 ts_tablet_manager.cc:1918] T 86337c23f495454f8649f5e1016d8b5f P e0643dd7e19b4fb5ad0df7418171dd52: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260620 11:42:47.447742 11658 log.cc:1196] T 86337c23f495454f8649f5e1016d8b5f P e0643dd7e19b4fb5ad0df7418171dd52: Deleting WAL directory at /tmp/dist-test-task1JRg7N/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781955746769051-7083-0/minicluster-data/ts-2/wal/wals/86337c23f495454f8649f5e1016d8b5f
I20260620 11:42:47.448019 11658 ts_tablet_manager.cc:1939] T 86337c23f495454f8649f5e1016d8b5f P e0643dd7e19b4fb5ad0df7418171dd52: Deleting consensus metadata
I20260620 11:42:47.448212 11659 ts_tablet_manager.cc:1918] T 86337c23f495454f8649f5e1016d8b5f P 7c824e5a1c7e4448bbd40d6bf8c5c000: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260620 11:42:47.448330 11659 log.cc:1196] T 86337c23f495454f8649f5e1016d8b5f P 7c824e5a1c7e4448bbd40d6bf8c5c000: Deleting WAL directory at /tmp/dist-test-task1JRg7N/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781955746769051-7083-0/minicluster-data/ts-1/wal/wals/86337c23f495454f8649f5e1016d8b5f
I20260620 11:42:47.448426 11150 catalog_manager.cc:4841] TS e0643dd7e19b4fb5ad0df7418171dd52 (127.6.234.195:37491): tablet 86337c23f495454f8649f5e1016d8b5f (table test-table [id=358e02c07bb34f619b13493f70ba6514]) successfully deleted
I20260620 11:42:47.448710 11659 ts_tablet_manager.cc:1939] T 86337c23f495454f8649f5e1016d8b5f P 7c824e5a1c7e4448bbd40d6bf8c5c000: Deleting consensus metadata
I20260620 11:42:47.449163 11149 catalog_manager.cc:4841] TS 7c824e5a1c7e4448bbd40d6bf8c5c000 (127.6.234.194:40509): tablet 86337c23f495454f8649f5e1016d8b5f (table test-table [id=358e02c07bb34f619b13493f70ba6514]) successfully deleted
W20260620 11:42:47.452014 11150 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.6.234.193:41431 (error 108)
I20260620 11:42:47.452503 11629 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 11:42:47.452387 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.234.193:41431 (local address 127.0.0.1:45196)
0620 11:42:47.452437 (+ 50us) negotiation.cc:107] Waiting for socket to connect
0620 11:42:47.452470 (+ 33us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.234.193:41431: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":30}
W20260620 11:42:47.452611 11150 catalog_manager.cc:4584] TS 66d73b1e343f4d55bafe247a62a894da (127.6.234.193:41431): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 86337c23f495454f8649f5e1016d8b5f: Network error: Client connection negotiation failed: client connection to 127.6.234.193:41431: connect: Connection refused (error 111)
I20260620 11:42:47.452625 7083 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task1JRg7N/build/release/bin/kudu with pid 11359
I20260620 11:42:47.457831 7083 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task1JRg7N/build/release/bin/kudu with pid 11492
I20260620 11:42:47.462579 7083 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task1JRg7N/build/release/bin/kudu with pid 11133
2026-06-20T11:42:47Z chronyd exiting
[ OK ] SecurityITest.TestDisableWebUI (1980 ms)
[ RUN ] SecurityITest.TestDisableAuthenticationEncryption
2026-06-20T11:42:47Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-20T11:42:47Z Disabled control of system clock
I20260620 11:42:47.489234 7083 external_mini_cluster.cc:1326] Running /tmp/dist-test-task1JRg7N/build/release/bin/kudu
/tmp/dist-test-task1JRg7N/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task1JRg7N/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781955746769051-7083-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task1JRg7N/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781955746769051-7083-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task1JRg7N/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781955746769051-7083-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task1JRg7N/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781955746769051-7083-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.6.234.254:45257
--webserver_interface=127.6.234.254
--webserver_port=0
--builtin_ntp_servers=127.6.234.212:41423
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.6.234.254:45257
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--rpc_authentication=disabled
--rpc_encryption=disabled with env {}
W20260620 11:42:47.570626 11664 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 11:42:47.570838 11664 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 11:42:47.570859 11664 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 11:42:47.571918 11664 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 11:42:47.571952 11664 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 11:42:47.571965 11664 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 11:42:47.571977 11664 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 11:42:47.571990 11664 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 11:42:47.573323 11664 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.6.234.212:41423
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task1JRg7N/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781955746769051-7083-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task1JRg7N/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781955746769051-7083-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.6.234.254:45257
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.6.234.254:45257
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task1JRg7N/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781955746769051-7083-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.6.234.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.11664
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task1JRg7N/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781955746769051-7083-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 20 Jun 2026 10:32:37 UTC on ccycloud.quasar-bgyofv.root.comops.site
build id 4457173
I20260620 11:42:47.573518 11664 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 11:42:47.573802 11664 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 11:42:47.578536 11669 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 11:42:47.578655 11664 server_base.cc:1034] running on GCE node
W20260620 11:42:47.578584 11672 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 11:42:47.578599 11670 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 11:42:47.578958 11664 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 11:42:47.579201 11664 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 11:42:47.580356 11664 hybrid_clock.cc:648] HybridClock initialized: now 1781955767580324 us; error 50 us; skew 500 ppm
I20260620 11:42:47.581576 11664 webserver.cc:460] Webserver started at http://127.6.234.254:46807/ using document root <none> and password file <none>
I20260620 11:42:47.581853 11664 fs_manager.cc:359] Metadata directory not provided
I20260620 11:42:47.581902 11664 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 11:42:47.582049 11664 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 11:42:47.583140 11664 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task1JRg7N/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781955746769051-7083-0/minicluster-data/master-0/data/instance:
uuid: "0a05dc3731924f579e3faec7c677c80c"
format_stamp: "Formatted at 2026-06-20 11:42:47 on dist-test-slave-cjc6"
server_key: "c618a5ea8aee0740edbc8596de335dbd"
server_key_iv: "68d51a87fd38a29d4c9f83e7426128d8"
server_key_version: "encryptionkey@0"
I20260620 11:42:47.583518 11664 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task1JRg7N/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781955746769051-7083-0/minicluster-data/master-0/wal/instance:
uuid: "0a05dc3731924f579e3faec7c677c80c"
format_stamp: "Formatted at 2026-06-20 11:42:47 on dist-test-slave-cjc6"
server_key: "c618a5ea8aee0740edbc8596de335dbd"
server_key_iv: "68d51a87fd38a29d4c9f83e7426128d8"
server_key_version: "encryptionkey@0"
I20260620 11:42:47.584782 11664 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260620 11:42:47.585521 11678 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 11:42:47.585760 11664 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260620 11:42:47.585860 11664 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task1JRg7N/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781955746769051-7083-0/minicluster-data/master-0/data,/tmp/dist-test-task1JRg7N/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781955746769051-7083-0/minicluster-data/master-0/wal
uuid: "0a05dc3731924f579e3faec7c677c80c"
format_stamp: "Formatted at 2026-06-20 11:42:47 on dist-test-slave-cjc6"
server_key: "c618a5ea8aee0740edbc8596de335dbd"
server_key_iv: "68d51a87fd38a29d4c9f83e7426128d8"
server_key_version: "encryptionkey@0"
I20260620 11:42:47.585990 11664 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task1JRg7N/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781955746769051-7083-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task1JRg7N/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781955746769051-7083-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task1JRg7N/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781955746769051-7083-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 11:42:47.589183 11664 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 11:42:47.589452 11664 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 11:42:47.589592 11664 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 11:42:47.593390 11664 rpc_server.cc:307] RPC server started. Bound to: 127.6.234.254:45257
I20260620 11:42:47.593429 11730 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.6.234.254:45257 every 8 connection(s)
I20260620 11:42:47.593809 11664 server_base.cc:1166] Dumped server information to /tmp/dist-test-task1JRg7N/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781955746769051-7083-0/minicluster-data/master-0/data/info.pb
I20260620 11:42:47.594449 11731 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 11:42:47.597054 11731 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 0a05dc3731924f579e3faec7c677c80c: Bootstrap starting.
I20260620 11:42:47.597556 11731 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 0a05dc3731924f579e3faec7c677c80c: Neither blocks nor log segments found. Creating new log.
I20260620 11:42:47.597885 11731 log.cc:824] T 00000000000000000000000000000000 P 0a05dc3731924f579e3faec7c677c80c: Log is configured to *not* fsync() on all Append() calls
I20260620 11:42:47.598471 11731 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 0a05dc3731924f579e3faec7c677c80c: No bootstrap required, opened a new log
I20260620 11:42:47.600018 11731 raft_consensus.cc:348] T 00000000000000000000000000000000 P 0a05dc3731924f579e3faec7c677c80c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0a05dc3731924f579e3faec7c677c80c" member_type: VOTER last_known_addr { host: "127.6.234.254" port: 45257 } }
I20260620 11:42:47.600144 11731 raft_consensus.cc:374] T 00000000000000000000000000000000 P 0a05dc3731924f579e3faec7c677c80c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 11:42:47.600167 11731 raft_consensus.cc:729] T 00000000000000000000000000000000 P 0a05dc3731924f579e3faec7c677c80c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0a05dc3731924f579e3faec7c677c80c, State: Initialized, Role: FOLLOWER
I20260620 11:42:47.600307 11731 consensus_queue.cc:260] T 00000000000000000000000000000000 P 0a05dc3731924f579e3faec7c677c80c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0a05dc3731924f579e3faec7c677c80c" member_type: VOTER last_known_addr { host: "127.6.234.254" port: 45257 } }
I20260620 11:42:47.600384 11731 raft_consensus.cc:388] T 00000000000000000000000000000000 P 0a05dc3731924f579e3faec7c677c80c [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260620 11:42:47.600406 11731 raft_consensus.cc:482] T 00000000000000000000000000000000 P 0a05dc3731924f579e3faec7c677c80c [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260620 11:42:47.600433 11731 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 0a05dc3731924f579e3faec7c677c80c [term 0 FOLLOWER]: Advancing to term 1
I20260620 11:42:47.600907 11731 raft_consensus.cc:504] T 00000000000000000000000000000000 P 0a05dc3731924f579e3faec7c677c80c [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0a05dc3731924f579e3faec7c677c80c" member_type: VOTER last_known_addr { host: "127.6.234.254" port: 45257 } }
I20260620 11:42:47.600988 11731 leader_election.cc:302] T 00000000000000000000000000000000 P 0a05dc3731924f579e3faec7c677c80c [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 0a05dc3731924f579e3faec7c677c80c; no voters:
I20260620 11:42:47.601112 11731 leader_election.cc:288] T 00000000000000000000000000000000 P 0a05dc3731924f579e3faec7c677c80c [CANDIDATE]: Term 1 election: Requested vote from peers
I20260620 11:42:47.601190 11734 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 0a05dc3731924f579e3faec7c677c80c [term 1 FOLLOWER]: Leader election won for term 1
I20260620 11:42:47.601358 11734 raft_consensus.cc:686] T 00000000000000000000000000000000 P 0a05dc3731924f579e3faec7c677c80c [term 1 LEADER]: Becoming Leader. State: Replica: 0a05dc3731924f579e3faec7c677c80c, State: Running, Role: LEADER
I20260620 11:42:47.601383 11731 sys_catalog.cc:564] T 00000000000000000000000000000000 P 0a05dc3731924f579e3faec7c677c80c [sys.catalog]: configured and running, proceeding with master startup.
I20260620 11:42:47.601449 11734 consensus_queue.cc:237] T 00000000000000000000000000000000 P 0a05dc3731924f579e3faec7c677c80c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0a05dc3731924f579e3faec7c677c80c" member_type: VOTER last_known_addr { host: "127.6.234.254" port: 45257 } }
I20260620 11:42:47.601831 11736 sys_catalog.cc:455] T 00000000000000000000000000000000 P 0a05dc3731924f579e3faec7c677c80c [sys.catalog]: SysCatalogTable state changed. Reason: New leader 0a05dc3731924f579e3faec7c677c80c. Latest consensus state: current_term: 1 leader_uuid: "0a05dc3731924f579e3faec7c677c80c" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0a05dc3731924f579e3faec7c677c80c" member_type: VOTER last_known_addr { host: "127.6.234.254" port: 45257 } } }
I20260620 11:42:47.601918 11736 sys_catalog.cc:458] T 00000000000000000000000000000000 P 0a05dc3731924f579e3faec7c677c80c [sys.catalog]: This master's current role is: LEADER
I20260620 11:42:47.602161 11735 sys_catalog.cc:455] T 00000000000000000000000000000000 P 0a05dc3731924f579e3faec7c677c80c [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "0a05dc3731924f579e3faec7c677c80c" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0a05dc3731924f579e3faec7c677c80c" member_type: VOTER last_known_addr { host: "127.6.234.254" port: 45257 } } }
I20260620 11:42:47.602308 11735 sys_catalog.cc:458] T 00000000000000000000000000000000 P 0a05dc3731924f579e3faec7c677c80c [sys.catalog]: This master's current role is: LEADER
I20260620 11:42:47.602355 11741 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260620 11:42:47.602880 11741 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260620 11:42:47.603333 7083 external_mini_cluster.cc:1388] Started /tmp/dist-test-task1JRg7N/build/release/bin/kudu as pid 11664
I20260620 11:42:47.603447 7083 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task1JRg7N/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781955746769051-7083-0/minicluster-data/master-0/wal/instance
I20260620 11:42:47.603610 7083 external_mini_cluster.cc:1428] Setting key ec328fc0a0c42d6ac796afbcf4197797
I20260620 11:42:47.604396 11741 catalog_manager.cc:1347] Generated new cluster ID: 03bbec40b9c74ed9960f93da7db8e4c1
I20260620 11:42:47.604451 11741 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260620 11:42:47.605984 11754 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 11:42:47.604242 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50268 (local address 127.6.234.254:45257)
0620 11:42:47.604462 (+ 220us) server_negotiation.cc:206] Beginning negotiation
0620 11:42:47.604471 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0620 11:42:47.604504 (+ 33us) server_negotiation.cc:407] Connection header received
0620 11:42:47.604613 (+ 109us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 11:42:47.604619 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 11:42:47.604652 (+ 33us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 11:42:47.604788 (+ 136us) server_negotiation.cc:226] Negotiated authn=SASL
0620 11:42:47.605499 (+ 711us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 11:42:47.605502 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 11:42:47.605503 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 11:42:47.605507 (+ 4us) server_negotiation.cc:922] calling sasl_server_start()
0620 11:42:47.605559 (+ 52us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 11:42:47.605571 (+ 12us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 11:42:47.605638 (+ 67us) server_negotiation.cc:1035] Waiting for connection context
0620 11:42:47.605738 (+ 100us) server_negotiation.cc:299] Negotiation successful
0620 11:42:47.605797 (+ 59us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":131,"thread_start_us":81,"threads_started":1}
I20260620 11:42:47.619079 11741 catalog_manager.cc:1370] Generated new certificate authority record
I20260620 11:42:47.619261 11741 catalog_manager.cc:1504] Loading token signing keys...
I20260620 11:42:47.628316 11741 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 0a05dc3731924f579e3faec7c677c80c: Generated new TSK 0
I20260620 11:42:47.628554 11741 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260620 11:42:47.632551 7083 external_mini_cluster.cc:1326] Running /tmp/dist-test-task1JRg7N/build/release/bin/kudu
/tmp/dist-test-task1JRg7N/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task1JRg7N/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781955746769051-7083-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-task1JRg7N/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781955746769051-7083-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task1JRg7N/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781955746769051-7083-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-task1JRg7N/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781955746769051-7083-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.6.234.193:0
--local_ip_for_outbound_sockets=127.6.234.193
--webserver_interface=127.6.234.193
--webserver_port=0
--tserver_master_addrs=127.6.234.254:45257
--builtin_ntp_servers=127.6.234.212:41423
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled
--rpc_encryption=disabled with env {}
W20260620 11:42:47.714287 11755 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 11:42:47.714509 11755 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 11:42:47.714545 11755 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 11:42:47.715680 11755 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 11:42:47.715757 11755 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 11:42:47.715806 11755 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.6.234.193
I20260620 11:42:47.717142 11755 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.6.234.212:41423
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task1JRg7N/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781955746769051-7083-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-task1JRg7N/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781955746769051-7083-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.6.234.193:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task1JRg7N/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781955746769051-7083-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.6.234.193
--webserver_port=0
--tserver_master_addrs=127.6.234.254:45257
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.11755
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.6.234.193
--log_dir=/tmp/dist-test-task1JRg7N/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781955746769051-7083-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 20 Jun 2026 10:32:37 UTC on ccycloud.quasar-bgyofv.root.comops.site
build id 4457173
I20260620 11:42:47.717404 11755 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 11:42:47.717737 11755 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 11:42:47.723414 11763 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 11:42:47.723610 11755 server_base.cc:1034] running on GCE node
W20260620 11:42:47.723426 11761 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 11:42:47.723456 11760 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 11:42:47.724021 11755 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 11:42:47.724262 11755 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 11:42:47.725436 11755 hybrid_clock.cc:648] HybridClock initialized: now 1781955767725400 us; error 52 us; skew 500 ppm
I20260620 11:42:47.726711 11755 webserver.cc:460] Webserver started at http://127.6.234.193:45353/ using document root <none> and password file <none>
I20260620 11:42:47.726964 11755 fs_manager.cc:359] Metadata directory not provided
I20260620 11:42:47.727031 11755 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 11:42:47.727182 11755 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 11:42:47.728237 11755 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task1JRg7N/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781955746769051-7083-0/minicluster-data/ts-0/data/instance:
uuid: "7ada3c0dc1074859bdb2dcc9a582a460"
format_stamp: "Formatted at 2026-06-20 11:42:47 on dist-test-slave-cjc6"
server_key: "00f3646fd8e987caa79798a6353c2066"
server_key_iv: "f4d674bb344dec77efc4d13dc0525391"
server_key_version: "encryptionkey@0"
I20260620 11:42:47.728631 11755 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task1JRg7N/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781955746769051-7083-0/minicluster-data/ts-0/wal/instance:
uuid: "7ada3c0dc1074859bdb2dcc9a582a460"
format_stamp: "Formatted at 2026-06-20 11:42:47 on dist-test-slave-cjc6"
server_key: "00f3646fd8e987caa79798a6353c2066"
server_key_iv: "f4d674bb344dec77efc4d13dc0525391"
server_key_version: "encryptionkey@0"
I20260620 11:42:47.729972 11755 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.003s
I20260620 11:42:47.730692 11769 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 11:42:47.730996 11755 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.000s
I20260620 11:42:47.731092 11755 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task1JRg7N/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781955746769051-7083-0/minicluster-data/ts-0/data,/tmp/dist-test-task1JRg7N/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781955746769051-7083-0/minicluster-data/ts-0/wal
uuid: "7ada3c0dc1074859bdb2dcc9a582a460"
format_stamp: "Formatted at 2026-06-20 11:42:47 on dist-test-slave-cjc6"
server_key: "00f3646fd8e987caa79798a6353c2066"
server_key_iv: "f4d674bb344dec77efc4d13dc0525391"
server_key_version: "encryptionkey@0"
I20260620 11:42:47.731163 11755 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task1JRg7N/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781955746769051-7083-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-task1JRg7N/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781955746769051-7083-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-task1JRg7N/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781955746769051-7083-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 11:42:47.734459 11755 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 11:42:47.734704 11755 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 11:42:47.734843 11755 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 11:42:47.735044 11755 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 11:42:47.735327 11755 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 11:42:47.735383 11755 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 11:42:47.735421 11755 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 11:42:47.735450 11755 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 11:42:47.741170 11755 rpc_server.cc:307] RPC server started. Bound to: 127.6.234.193:38049
I20260620 11:42:47.741246 11882 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.6.234.193:38049 every 8 connection(s)
I20260620 11:42:47.741559 11755 server_base.cc:1166] Dumped server information to /tmp/dist-test-task1JRg7N/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781955746769051-7083-0/minicluster-data/ts-0/data/info.pb
I20260620 11:42:47.744452 11754 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 11:42:47.742351 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.6.234.193:50411 (local address 127.6.234.254:45257)
0620 11:42:47.742406 (+ 55us) server_negotiation.cc:206] Beginning negotiation
0620 11:42:47.742408 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 11:42:47.742609 (+ 201us) server_negotiation.cc:407] Connection header received
0620 11:42:47.743062 (+ 453us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 11:42:47.743063 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 11:42:47.743069 (+ 6us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 11:42:47.743131 (+ 62us) server_negotiation.cc:226] Negotiated authn=SASL
0620 11:42:47.744265 (+ 1134us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 11:42:47.744265 (+ 0us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 11:42:47.744266 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 11:42:47.744268 (+ 2us) server_negotiation.cc:922] calling sasl_server_start()
0620 11:42:47.744278 (+ 10us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 11:42:47.744281 (+ 3us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 11:42:47.744326 (+ 45us) server_negotiation.cc:1035] Waiting for connection context
0620 11:42:47.744389 (+ 63us) server_negotiation.cc:299] Negotiation successful
0620 11:42:47.744402 (+ 13us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":32}
I20260620 11:42:47.744625 11885 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 11:42:47.742300 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.234.254:45257 (local address 127.6.234.193:50411)
0620 11:42:47.742484 (+ 184us) negotiation.cc:107] Waiting for socket to connect
0620 11:42:47.742522 (+ 38us) client_negotiation.cc:174] Beginning negotiation
0620 11:42:47.742983 (+ 461us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 11:42:47.743183 (+ 200us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 11:42:47.743186 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 11:42:47.743194 (+ 8us) client_negotiation.cc:189] Negotiated authn=SASL
0620 11:42:47.744140 (+ 946us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0620 11:42:47.744140 (+ 0us) client_negotiation.cc:656] calling sasl_client_start()
0620 11:42:47.744168 (+ 28us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0620 11:42:47.744195 (+ 27us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 11:42:47.744324 (+ 129us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 11:42:47.744326 (+ 2us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 11:42:47.744328 (+ 2us) client_negotiation.cc:769] Sending connection context
0620 11:42:47.744386 (+ 58us) client_negotiation.cc:240] Negotiation successful
0620 11:42:47.744431 (+ 45us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":122,"thread_start_us":94,"threads_started":1}
I20260620 11:42:47.745121 11883 heartbeater.cc:347] Connected to a master server at 127.6.234.254:45257
I20260620 11:42:47.745210 11883 heartbeater.cc:464] Registering TS with master...
I20260620 11:42:47.745319 11883 heartbeater.cc:511] Master 127.6.234.254:45257 requested a full tablet report, sending...
I20260620 11:42:47.745833 11695 ts_manager.cc:194] Registered new tserver with Master: 7ada3c0dc1074859bdb2dcc9a582a460 (127.6.234.193:38049)
I20260620 11:42:47.748826 7083 external_mini_cluster.cc:1388] Started /tmp/dist-test-task1JRg7N/build/release/bin/kudu as pid 11755
I20260620 11:42:47.748910 7083 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task1JRg7N/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781955746769051-7083-0/minicluster-data/ts-0/wal/instance
I20260620 11:42:47.749039 7083 external_mini_cluster.cc:1428] Setting key 2ad94e45f2c3ade08dbdb28c1f160a4c
I20260620 11:42:47.749965 7083 external_mini_cluster.cc:1326] Running /tmp/dist-test-task1JRg7N/build/release/bin/kudu
/tmp/dist-test-task1JRg7N/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task1JRg7N/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781955746769051-7083-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-task1JRg7N/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781955746769051-7083-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task1JRg7N/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781955746769051-7083-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-task1JRg7N/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781955746769051-7083-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.6.234.194:0
--local_ip_for_outbound_sockets=127.6.234.194
--webserver_interface=127.6.234.194
--webserver_port=0
--tserver_master_addrs=127.6.234.254:45257
--builtin_ntp_servers=127.6.234.212:41423
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled
--rpc_encryption=disabled with env {}
W20260620 11:42:47.830617 11886 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 11:42:47.830824 11886 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 11:42:47.830847 11886 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 11:42:47.831931 11886 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 11:42:47.831965 11886 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 11:42:47.831990 11886 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.6.234.194
I20260620 11:42:47.833228 11886 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.6.234.212:41423
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task1JRg7N/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781955746769051-7083-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-task1JRg7N/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781955746769051-7083-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.6.234.194:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task1JRg7N/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781955746769051-7083-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.6.234.194
--webserver_port=0
--tserver_master_addrs=127.6.234.254:45257
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.11886
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.6.234.194
--log_dir=/tmp/dist-test-task1JRg7N/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781955746769051-7083-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 20 Jun 2026 10:32:37 UTC on ccycloud.quasar-bgyofv.root.comops.site
build id 4457173
I20260620 11:42:47.833508 11886 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 11:42:47.833817 11886 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260620 11:42:47.838972 11886 server_base.cc:1034] running on GCE node
W20260620 11:42:47.838943 11891 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 11:42:47.838943 11894 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 11:42:47.838923 11892 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 11:42:47.839359 11886 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 11:42:47.839614 11886 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 11:42:47.840752 11886 hybrid_clock.cc:648] HybridClock initialized: now 1781955767840725 us; error 42 us; skew 500 ppm
I20260620 11:42:47.841997 11886 webserver.cc:460] Webserver started at http://127.6.234.194:43133/ using document root <none> and password file <none>
I20260620 11:42:47.842228 11886 fs_manager.cc:359] Metadata directory not provided
I20260620 11:42:47.842278 11886 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 11:42:47.842440 11886 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 11:42:47.843463 11886 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task1JRg7N/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781955746769051-7083-0/minicluster-data/ts-1/data/instance:
uuid: "84b9816672b94db488d7a779f4e2b8e6"
format_stamp: "Formatted at 2026-06-20 11:42:47 on dist-test-slave-cjc6"
server_key: "c24365cb75c43118481724a004ce2cd9"
server_key_iv: "d6f3f6eca1cd185bde9433074b7262da"
server_key_version: "encryptionkey@0"
I20260620 11:42:47.843823 11886 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task1JRg7N/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781955746769051-7083-0/minicluster-data/ts-1/wal/instance:
uuid: "84b9816672b94db488d7a779f4e2b8e6"
format_stamp: "Formatted at 2026-06-20 11:42:47 on dist-test-slave-cjc6"
server_key: "c24365cb75c43118481724a004ce2cd9"
server_key_iv: "d6f3f6eca1cd185bde9433074b7262da"
server_key_version: "encryptionkey@0"
I20260620 11:42:47.845077 11886 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260620 11:42:47.845836 11900 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 11:42:47.845992 11886 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260620 11:42:47.846071 11886 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task1JRg7N/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781955746769051-7083-0/minicluster-data/ts-1/data,/tmp/dist-test-task1JRg7N/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781955746769051-7083-0/minicluster-data/ts-1/wal
uuid: "84b9816672b94db488d7a779f4e2b8e6"
format_stamp: "Formatted at 2026-06-20 11:42:47 on dist-test-slave-cjc6"
server_key: "c24365cb75c43118481724a004ce2cd9"
server_key_iv: "d6f3f6eca1cd185bde9433074b7262da"
server_key_version: "encryptionkey@0"
I20260620 11:42:47.846122 11886 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task1JRg7N/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781955746769051-7083-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-task1JRg7N/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781955746769051-7083-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-task1JRg7N/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781955746769051-7083-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 11:42:47.849764 11886 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 11:42:47.850036 11886 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 11:42:47.850129 11886 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 11:42:47.850283 11886 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 11:42:47.850512 11886 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 11:42:47.850543 11886 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 11:42:47.850561 11886 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 11:42:47.850574 11886 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 11:42:47.855872 11886 rpc_server.cc:307] RPC server started. Bound to: 127.6.234.194:33253
I20260620 11:42:47.855948 12013 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.6.234.194:33253 every 8 connection(s)
I20260620 11:42:47.856271 11886 server_base.cc:1166] Dumped server information to /tmp/dist-test-task1JRg7N/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781955746769051-7083-0/minicluster-data/ts-1/data/info.pb
I20260620 11:42:47.859135 11754 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 11:42:47.857047 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.6.234.194:52387 (local address 127.6.234.254:45257)
0620 11:42:47.857114 (+ 67us) server_negotiation.cc:206] Beginning negotiation
0620 11:42:47.857116 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 11:42:47.857320 (+ 204us) server_negotiation.cc:407] Connection header received
0620 11:42:47.857790 (+ 470us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 11:42:47.857791 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 11:42:47.857796 (+ 5us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 11:42:47.857860 (+ 64us) server_negotiation.cc:226] Negotiated authn=SASL
0620 11:42:47.858943 (+ 1083us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 11:42:47.858945 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 11:42:47.858945 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 11:42:47.858949 (+ 4us) server_negotiation.cc:922] calling sasl_server_start()
0620 11:42:47.858963 (+ 14us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 11:42:47.858967 (+ 4us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 11:42:47.859016 (+ 49us) server_negotiation.cc:1035] Waiting for connection context
0620 11:42:47.859070 (+ 54us) server_negotiation.cc:299] Negotiation successful
0620 11:42:47.859081 (+ 11us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":34}
I20260620 11:42:47.859297 12016 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 11:42:47.857005 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.234.254:45257 (local address 127.6.234.194:52387)
0620 11:42:47.857188 (+ 183us) negotiation.cc:107] Waiting for socket to connect
0620 11:42:47.857226 (+ 38us) client_negotiation.cc:174] Beginning negotiation
0620 11:42:47.857658 (+ 432us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 11:42:47.857921 (+ 263us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 11:42:47.857924 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 11:42:47.857932 (+ 8us) client_negotiation.cc:189] Negotiated authn=SASL
0620 11:42:47.858788 (+ 856us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0620 11:42:47.858788 (+ 0us) client_negotiation.cc:656] calling sasl_client_start()
0620 11:42:47.858831 (+ 43us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0620 11:42:47.858858 (+ 27us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 11:42:47.859012 (+ 154us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 11:42:47.859014 (+ 2us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 11:42:47.859016 (+ 2us) client_negotiation.cc:769] Sending connection context
0620 11:42:47.859067 (+ 51us) client_negotiation.cc:240] Negotiation successful
0620 11:42:47.859110 (+ 43us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":120,"thread_start_us":93,"threads_started":1}
I20260620 11:42:47.859766 12014 heartbeater.cc:347] Connected to a master server at 127.6.234.254:45257
I20260620 11:42:47.859856 12014 heartbeater.cc:464] Registering TS with master...
I20260620 11:42:47.859966 12014 heartbeater.cc:511] Master 127.6.234.254:45257 requested a full tablet report, sending...
I20260620 11:42:47.860287 11695 ts_manager.cc:194] Registered new tserver with Master: 84b9816672b94db488d7a779f4e2b8e6 (127.6.234.194:33253)
I20260620 11:42:47.864319 7083 external_mini_cluster.cc:1388] Started /tmp/dist-test-task1JRg7N/build/release/bin/kudu as pid 11886
I20260620 11:42:47.864425 7083 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task1JRg7N/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781955746769051-7083-0/minicluster-data/ts-1/wal/instance
I20260620 11:42:47.864607 7083 external_mini_cluster.cc:1428] Setting key e8694fe15fee1b32623d0e8a2ee406f3
I20260620 11:42:47.865465 7083 external_mini_cluster.cc:1326] Running /tmp/dist-test-task1JRg7N/build/release/bin/kudu
/tmp/dist-test-task1JRg7N/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task1JRg7N/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781955746769051-7083-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-task1JRg7N/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781955746769051-7083-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task1JRg7N/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781955746769051-7083-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-task1JRg7N/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781955746769051-7083-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.6.234.195:0
--local_ip_for_outbound_sockets=127.6.234.195
--webserver_interface=127.6.234.195
--webserver_port=0
--tserver_master_addrs=127.6.234.254:45257
--builtin_ntp_servers=127.6.234.212:41423
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled
--rpc_encryption=disabled with env {}
W20260620 11:42:47.945148 12017 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 11:42:47.945353 12017 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 11:42:47.945377 12017 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 11:42:47.946584 12017 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 11:42:47.946628 12017 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 11:42:47.946655 12017 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.6.234.195
I20260620 11:42:47.948001 12017 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.6.234.212:41423
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task1JRg7N/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781955746769051-7083-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-task1JRg7N/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781955746769051-7083-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.6.234.195:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task1JRg7N/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781955746769051-7083-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.6.234.195
--webserver_port=0
--tserver_master_addrs=127.6.234.254:45257
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.12017
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.6.234.195
--log_dir=/tmp/dist-test-task1JRg7N/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781955746769051-7083-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 20 Jun 2026 10:32:37 UTC on ccycloud.quasar-bgyofv.root.comops.site
build id 4457173
I20260620 11:42:47.948246 12017 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 11:42:47.948575 12017 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 11:42:47.953570 12023 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 11:42:47.953570 12022 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 11:42:47.953634 12017 server_base.cc:1034] running on GCE node
W20260620 11:42:47.953613 12025 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 11:42:47.954048 12017 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 11:42:47.954295 12017 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 11:42:47.955458 12017 hybrid_clock.cc:648] HybridClock initialized: now 1781955767955440 us; error 46 us; skew 500 ppm
I20260620 11:42:47.956719 12017 webserver.cc:460] Webserver started at http://127.6.234.195:38099/ using document root <none> and password file <none>
I20260620 11:42:47.956991 12017 fs_manager.cc:359] Metadata directory not provided
I20260620 11:42:47.957062 12017 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 11:42:47.957216 12017 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 11:42:47.958305 12017 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task1JRg7N/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781955746769051-7083-0/minicluster-data/ts-2/data/instance:
uuid: "a2e137871e0648f4aaf774ff62871920"
format_stamp: "Formatted at 2026-06-20 11:42:47 on dist-test-slave-cjc6"
server_key: "c923cface8302d531e09483d38b1ae8b"
server_key_iv: "1f8018db9e9e7be796b0481671b12843"
server_key_version: "encryptionkey@0"
I20260620 11:42:47.958657 12017 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task1JRg7N/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781955746769051-7083-0/minicluster-data/ts-2/wal/instance:
uuid: "a2e137871e0648f4aaf774ff62871920"
format_stamp: "Formatted at 2026-06-20 11:42:47 on dist-test-slave-cjc6"
server_key: "c923cface8302d531e09483d38b1ae8b"
server_key_iv: "1f8018db9e9e7be796b0481671b12843"
server_key_version: "encryptionkey@0"
I20260620 11:42:47.960016 12017 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.001s sys 0.001s
I20260620 11:42:47.960819 12031 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 11:42:47.961066 12017 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260620 11:42:47.961165 12017 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task1JRg7N/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781955746769051-7083-0/minicluster-data/ts-2/data,/tmp/dist-test-task1JRg7N/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781955746769051-7083-0/minicluster-data/ts-2/wal
uuid: "a2e137871e0648f4aaf774ff62871920"
format_stamp: "Formatted at 2026-06-20 11:42:47 on dist-test-slave-cjc6"
server_key: "c923cface8302d531e09483d38b1ae8b"
server_key_iv: "1f8018db9e9e7be796b0481671b12843"
server_key_version: "encryptionkey@0"
I20260620 11:42:47.961246 12017 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task1JRg7N/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781955746769051-7083-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-task1JRg7N/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781955746769051-7083-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-task1JRg7N/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781955746769051-7083-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 11:42:47.964625 12017 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 11:42:47.964901 12017 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 11:42:47.965039 12017 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 11:42:47.965233 12017 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 11:42:47.965533 12017 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 11:42:47.965583 12017 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 11:42:47.965622 12017 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 11:42:47.965647 12017 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 11:42:47.971228 12017 rpc_server.cc:307] RPC server started. Bound to: 127.6.234.195:42715
I20260620 11:42:47.971304 12144 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.6.234.195:42715 every 8 connection(s)
I20260620 11:42:47.971628 12017 server_base.cc:1166] Dumped server information to /tmp/dist-test-task1JRg7N/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781955746769051-7083-0/minicluster-data/ts-2/data/info.pb
I20260620 11:42:47.974561 11754 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 11:42:47.972375 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.6.234.195:43867 (local address 127.6.234.254:45257)
0620 11:42:47.972443 (+ 68us) server_negotiation.cc:206] Beginning negotiation
0620 11:42:47.972445 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 11:42:47.972686 (+ 241us) server_negotiation.cc:407] Connection header received
0620 11:42:47.973147 (+ 461us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 11:42:47.973148 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 11:42:47.973154 (+ 6us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 11:42:47.973213 (+ 59us) server_negotiation.cc:226] Negotiated authn=SASL
0620 11:42:47.974359 (+ 1146us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 11:42:47.974360 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 11:42:47.974361 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 11:42:47.974364 (+ 3us) server_negotiation.cc:922] calling sasl_server_start()
0620 11:42:47.974379 (+ 15us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 11:42:47.974383 (+ 4us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 11:42:47.974427 (+ 44us) server_negotiation.cc:1035] Waiting for connection context
0620 11:42:47.974495 (+ 68us) server_negotiation.cc:299] Negotiation successful
0620 11:42:47.974508 (+ 13us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":44}
I20260620 11:42:47.974733 12147 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 11:42:47.972366 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.234.254:45257 (local address 127.6.234.195:43867)
0620 11:42:47.972558 (+ 192us) negotiation.cc:107] Waiting for socket to connect
0620 11:42:47.972598 (+ 40us) client_negotiation.cc:174] Beginning negotiation
0620 11:42:47.973064 (+ 466us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 11:42:47.973265 (+ 201us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 11:42:47.973268 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 11:42:47.973276 (+ 8us) client_negotiation.cc:189] Negotiated authn=SASL
0620 11:42:47.974206 (+ 930us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0620 11:42:47.974207 (+ 1us) client_negotiation.cc:656] calling sasl_client_start()
0620 11:42:47.974237 (+ 30us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0620 11:42:47.974267 (+ 30us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 11:42:47.974431 (+ 164us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 11:42:47.974434 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 11:42:47.974436 (+ 2us) client_negotiation.cc:769] Sending connection context
0620 11:42:47.974489 (+ 53us) client_negotiation.cc:240] Negotiation successful
0620 11:42:47.974536 (+ 47us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":129,"thread_start_us":83,"threads_started":1}
I20260620 11:42:47.975231 12145 heartbeater.cc:347] Connected to a master server at 127.6.234.254:45257
I20260620 11:42:47.975319 12145 heartbeater.cc:464] Registering TS with master...
I20260620 11:42:47.975426 12145 heartbeater.cc:511] Master 127.6.234.254:45257 requested a full tablet report, sending...
I20260620 11:42:47.975732 11695 ts_manager.cc:194] Registered new tserver with Master: a2e137871e0648f4aaf774ff62871920 (127.6.234.195:42715)
I20260620 11:42:47.980384 7083 external_mini_cluster.cc:1388] Started /tmp/dist-test-task1JRg7N/build/release/bin/kudu as pid 12017
I20260620 11:42:47.980506 7083 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task1JRg7N/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781955746769051-7083-0/minicluster-data/ts-2/wal/instance
I20260620 11:42:47.980687 7083 external_mini_cluster.cc:1428] Setting key e309e586c21a077934236217129b84a1
I20260620 11:42:47.981594 7083 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260620 11:42:47.984154 11754 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 11:42:47.982971 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50278 (local address 127.6.234.254:45257)
0620 11:42:47.983011 (+ 40us) server_negotiation.cc:206] Beginning negotiation
0620 11:42:47.983013 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 11:42:47.983138 (+ 125us) server_negotiation.cc:407] Connection header received
0620 11:42:47.983219 (+ 81us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 11:42:47.983220 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 11:42:47.983225 (+ 5us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 11:42:47.983294 (+ 69us) server_negotiation.cc:226] Negotiated authn=SASL
0620 11:42:47.983922 (+ 628us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 11:42:47.983924 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 11:42:47.983924 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 11:42:47.983929 (+ 5us) server_negotiation.cc:922] calling sasl_server_start()
0620 11:42:47.983945 (+ 16us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 11:42:47.983950 (+ 5us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 11:42:47.984007 (+ 57us) server_negotiation.cc:1035] Waiting for connection context
0620 11:42:47.984064 (+ 57us) server_negotiation.cc:299] Negotiation successful
0620 11:42:47.984081 (+ 17us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":17}
I20260620 11:42:47.985754 11695 catalog_manager.cc:1908] Servicing CreateTable request from {username='slave'} at 127.0.0.1:50278:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260620 11:42:47.987304 11695 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260620 11:42:47.990929 12153 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 11:42:47.989476 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.234.193:38049 (local address 127.0.0.1:54580)
0620 11:42:47.989593 (+ 117us) negotiation.cc:107] Waiting for socket to connect
0620 11:42:47.989632 (+ 39us) client_negotiation.cc:174] Beginning negotiation
0620 11:42:47.989682 (+ 50us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 11:42:47.990015 (+ 333us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 11:42:47.990020 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 11:42:47.990026 (+ 6us) client_negotiation.cc:189] Negotiated authn=SASL
0620 11:42:47.990491 (+ 465us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0620 11:42:47.990491 (+ 0us) client_negotiation.cc:656] calling sasl_client_start()
0620 11:42:47.990505 (+ 14us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0620 11:42:47.990522 (+ 17us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 11:42:47.990759 (+ 237us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 11:42:47.990765 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 11:42:47.990768 (+ 3us) client_negotiation.cc:769] Sending connection context
0620 11:42:47.990851 (+ 83us) client_negotiation.cc:240] Negotiation successful
0620 11:42:47.990865 (+ 14us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":94,"thread_start_us":74,"threads_started":1}
I20260620 11:42:47.990963 12155 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 11:42:47.989574 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54580 (local address 127.6.234.193:38049)
0620 11:42:47.989858 (+ 284us) server_negotiation.cc:206] Beginning negotiation
0620 11:42:47.989863 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 11:42:47.989875 (+ 12us) server_negotiation.cc:407] Connection header received
0620 11:42:47.989904 (+ 29us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 11:42:47.989906 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 11:42:47.989915 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 11:42:47.990015 (+ 100us) server_negotiation.cc:226] Negotiated authn=SASL
0620 11:42:47.990604 (+ 589us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 11:42:47.990610 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 11:42:47.990611 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 11:42:47.990616 (+ 5us) server_negotiation.cc:922] calling sasl_server_start()
0620 11:42:47.990655 (+ 39us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 11:42:47.990680 (+ 25us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 11:42:47.990745 (+ 65us) server_negotiation.cc:1035] Waiting for connection context
0620 11:42:47.990867 (+ 122us) server_negotiation.cc:299] Negotiation successful
0620 11:42:47.990883 (+ 16us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":245,"thread_start_us":134,"threads_started":1}
I20260620 11:42:47.991448 12156 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 11:42:47.989589 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.234.195:42715 (local address 127.0.0.1:37058)
0620 11:42:47.989827 (+ 238us) negotiation.cc:107] Waiting for socket to connect
0620 11:42:47.989837 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0620 11:42:47.989874 (+ 37us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 11:42:47.990658 (+ 784us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 11:42:47.990658 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 11:42:47.990660 (+ 2us) client_negotiation.cc:189] Negotiated authn=SASL
0620 11:42:47.991073 (+ 413us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0620 11:42:47.991073 (+ 0us) client_negotiation.cc:656] calling sasl_client_start()
0620 11:42:47.991081 (+ 8us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0620 11:42:47.991094 (+ 13us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 11:42:47.991317 (+ 223us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 11:42:47.991318 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 11:42:47.991319 (+ 1us) client_negotiation.cc:769] Sending connection context
0620 11:42:47.991375 (+ 56us) client_negotiation.cc:240] Negotiation successful
0620 11:42:47.991389 (+ 14us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":221,"thread_start_us":76,"threads_started":1}
I20260620 11:42:47.991468 12157 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 11:42:47.989819 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37058 (local address 127.6.234.195:42715)
0620 11:42:47.990508 (+ 689us) server_negotiation.cc:206] Beginning negotiation
0620 11:42:47.990512 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 11:42:47.990520 (+ 8us) server_negotiation.cc:407] Connection header received
0620 11:42:47.990541 (+ 21us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 11:42:47.990543 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 11:42:47.990573 (+ 30us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 11:42:47.990659 (+ 86us) server_negotiation.cc:226] Negotiated authn=SASL
0620 11:42:47.991184 (+ 525us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 11:42:47.991191 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 11:42:47.991191 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 11:42:47.991195 (+ 4us) server_negotiation.cc:922] calling sasl_server_start()
0620 11:42:47.991230 (+ 35us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 11:42:47.991251 (+ 21us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 11:42:47.991314 (+ 63us) server_negotiation.cc:1035] Waiting for connection context
0620 11:42:47.991383 (+ 69us) server_negotiation.cc:299] Negotiation successful
0620 11:42:47.991397 (+ 14us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":654,"thread_start_us":81,"threads_started":1}
I20260620 11:42:47.992137 12154 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 11:42:47.989709 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.234.194:33253 (local address 127.0.0.1:44572)
0620 11:42:47.989990 (+ 281us) negotiation.cc:107] Waiting for socket to connect
0620 11:42:47.990000 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0620 11:42:47.990040 (+ 40us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 11:42:47.991366 (+ 1326us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 11:42:47.991367 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 11:42:47.991369 (+ 2us) client_negotiation.cc:189] Negotiated authn=SASL
0620 11:42:47.991778 (+ 409us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0620 11:42:47.991779 (+ 1us) client_negotiation.cc:656] calling sasl_client_start()
0620 11:42:47.991785 (+ 6us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0620 11:42:47.991797 (+ 12us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 11:42:47.992020 (+ 223us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 11:42:47.992021 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 11:42:47.992022 (+ 1us) client_negotiation.cc:769] Sending connection context
0620 11:42:47.992063 (+ 41us) client_negotiation.cc:240] Negotiation successful
0620 11:42:47.992076 (+ 13us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":258,"thread_start_us":79,"threads_started":1}
I20260620 11:42:47.992108 11817 tablet_service.cc:1467] Processing CreateTablet for tablet 0bfa55ce3788466c87a0fa6571939ff1 (DEFAULT_TABLE table=test-table [id=a0cda2612ce5495eb218c8fd7dd48b04]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 11:42:47.992318 12158 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 11:42:47.989933 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44572 (local address 127.6.234.194:33253)
0620 11:42:47.990105 (+ 172us) server_negotiation.cc:206] Beginning negotiation
0620 11:42:47.990110 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 11:42:47.990120 (+ 10us) server_negotiation.cc:407] Connection header received
0620 11:42:47.990987 (+ 867us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 11:42:47.990998 (+ 11us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 11:42:47.991008 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 11:42:47.991079 (+ 71us) server_negotiation.cc:226] Negotiated authn=SASL
0620 11:42:47.991880 (+ 801us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 11:42:47.991889 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 11:42:47.991890 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 11:42:47.991894 (+ 4us) server_negotiation.cc:922] calling sasl_server_start()
0620 11:42:47.991925 (+ 31us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 11:42:47.991949 (+ 24us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 11:42:47.992108 (+ 159us) server_negotiation.cc:1035] Waiting for connection context
0620 11:42:47.992132 (+ 24us) server_negotiation.cc:299] Negotiation successful
0620 11:42:47.992148 (+ 16us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":141,"thread_start_us":77,"threads_started":1}
I20260620 11:42:47.992475 11817 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 0bfa55ce3788466c87a0fa6571939ff1. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 11:42:47.992955 11948 tablet_service.cc:1467] Processing CreateTablet for tablet 0bfa55ce3788466c87a0fa6571939ff1 (DEFAULT_TABLE table=test-table [id=a0cda2612ce5495eb218c8fd7dd48b04]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 11:42:47.993284 11948 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 0bfa55ce3788466c87a0fa6571939ff1. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 11:42:47.994029 12079 tablet_service.cc:1467] Processing CreateTablet for tablet 0bfa55ce3788466c87a0fa6571939ff1 (DEFAULT_TABLE table=test-table [id=a0cda2612ce5495eb218c8fd7dd48b04]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 11:42:47.994354 12079 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 0bfa55ce3788466c87a0fa6571939ff1. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 11:42:47.995973 12159 tablet_bootstrap.cc:492] T 0bfa55ce3788466c87a0fa6571939ff1 P 7ada3c0dc1074859bdb2dcc9a582a460: Bootstrap starting.
I20260620 11:42:47.996501 12160 tablet_bootstrap.cc:492] T 0bfa55ce3788466c87a0fa6571939ff1 P 84b9816672b94db488d7a779f4e2b8e6: Bootstrap starting.
I20260620 11:42:47.996891 12159 tablet_bootstrap.cc:654] T 0bfa55ce3788466c87a0fa6571939ff1 P 7ada3c0dc1074859bdb2dcc9a582a460: Neither blocks nor log segments found. Creating new log.
I20260620 11:42:47.997201 12160 tablet_bootstrap.cc:654] T 0bfa55ce3788466c87a0fa6571939ff1 P 84b9816672b94db488d7a779f4e2b8e6: Neither blocks nor log segments found. Creating new log.
I20260620 11:42:47.997296 12159 log.cc:824] T 0bfa55ce3788466c87a0fa6571939ff1 P 7ada3c0dc1074859bdb2dcc9a582a460: Log is configured to *not* fsync() on all Append() calls
I20260620 11:42:47.997531 12160 log.cc:824] T 0bfa55ce3788466c87a0fa6571939ff1 P 84b9816672b94db488d7a779f4e2b8e6: Log is configured to *not* fsync() on all Append() calls
I20260620 11:42:47.997545 12161 tablet_bootstrap.cc:492] T 0bfa55ce3788466c87a0fa6571939ff1 P a2e137871e0648f4aaf774ff62871920: Bootstrap starting.
I20260620 11:42:47.998184 12161 tablet_bootstrap.cc:654] T 0bfa55ce3788466c87a0fa6571939ff1 P a2e137871e0648f4aaf774ff62871920: Neither blocks nor log segments found. Creating new log.
I20260620 11:42:47.998312 12160 tablet_bootstrap.cc:492] T 0bfa55ce3788466c87a0fa6571939ff1 P 84b9816672b94db488d7a779f4e2b8e6: No bootstrap required, opened a new log
I20260620 11:42:47.998353 12159 tablet_bootstrap.cc:492] T 0bfa55ce3788466c87a0fa6571939ff1 P 7ada3c0dc1074859bdb2dcc9a582a460: No bootstrap required, opened a new log
I20260620 11:42:47.998392 12160 ts_tablet_manager.cc:1397] T 0bfa55ce3788466c87a0fa6571939ff1 P 84b9816672b94db488d7a779f4e2b8e6: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260620 11:42:47.998445 12159 ts_tablet_manager.cc:1397] T 0bfa55ce3788466c87a0fa6571939ff1 P 7ada3c0dc1074859bdb2dcc9a582a460: Time spent bootstrapping tablet: real 0.003s user 0.002s sys 0.000s
I20260620 11:42:47.998521 12161 log.cc:824] T 0bfa55ce3788466c87a0fa6571939ff1 P a2e137871e0648f4aaf774ff62871920: Log is configured to *not* fsync() on all Append() calls
I20260620 11:42:47.999310 12161 tablet_bootstrap.cc:492] T 0bfa55ce3788466c87a0fa6571939ff1 P a2e137871e0648f4aaf774ff62871920: No bootstrap required, opened a new log
I20260620 11:42:47.999380 12161 ts_tablet_manager.cc:1397] T 0bfa55ce3788466c87a0fa6571939ff1 P a2e137871e0648f4aaf774ff62871920: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260620 11:42:48.000864 12159 raft_consensus.cc:348] T 0bfa55ce3788466c87a0fa6571939ff1 P 7ada3c0dc1074859bdb2dcc9a582a460 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7ada3c0dc1074859bdb2dcc9a582a460" member_type: VOTER last_known_addr { host: "127.6.234.193" port: 38049 } } peers { permanent_uuid: "84b9816672b94db488d7a779f4e2b8e6" member_type: VOTER last_known_addr { host: "127.6.234.194" port: 33253 } } peers { permanent_uuid: "a2e137871e0648f4aaf774ff62871920" member_type: VOTER last_known_addr { host: "127.6.234.195" port: 42715 } }
I20260620 11:42:48.001080 12159 raft_consensus.cc:374] T 0bfa55ce3788466c87a0fa6571939ff1 P 7ada3c0dc1074859bdb2dcc9a582a460 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 11:42:48.000985 12160 raft_consensus.cc:348] T 0bfa55ce3788466c87a0fa6571939ff1 P 84b9816672b94db488d7a779f4e2b8e6 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7ada3c0dc1074859bdb2dcc9a582a460" member_type: VOTER last_known_addr { host: "127.6.234.193" port: 38049 } } peers { permanent_uuid: "84b9816672b94db488d7a779f4e2b8e6" member_type: VOTER last_known_addr { host: "127.6.234.194" port: 33253 } } peers { permanent_uuid: "a2e137871e0648f4aaf774ff62871920" member_type: VOTER last_known_addr { host: "127.6.234.195" port: 42715 } }
I20260620 11:42:48.001130 12159 raft_consensus.cc:729] T 0bfa55ce3788466c87a0fa6571939ff1 P 7ada3c0dc1074859bdb2dcc9a582a460 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7ada3c0dc1074859bdb2dcc9a582a460, State: Initialized, Role: FOLLOWER
I20260620 11:42:48.001140 12160 raft_consensus.cc:374] T 0bfa55ce3788466c87a0fa6571939ff1 P 84b9816672b94db488d7a779f4e2b8e6 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 11:42:48.001173 12160 raft_consensus.cc:729] T 0bfa55ce3788466c87a0fa6571939ff1 P 84b9816672b94db488d7a779f4e2b8e6 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 84b9816672b94db488d7a779f4e2b8e6, State: Initialized, Role: FOLLOWER
I20260620 11:42:48.001282 12159 consensus_queue.cc:260] T 0bfa55ce3788466c87a0fa6571939ff1 P 7ada3c0dc1074859bdb2dcc9a582a460 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7ada3c0dc1074859bdb2dcc9a582a460" member_type: VOTER last_known_addr { host: "127.6.234.193" port: 38049 } } peers { permanent_uuid: "84b9816672b94db488d7a779f4e2b8e6" member_type: VOTER last_known_addr { host: "127.6.234.194" port: 33253 } } peers { permanent_uuid: "a2e137871e0648f4aaf774ff62871920" member_type: VOTER last_known_addr { host: "127.6.234.195" port: 42715 } }
I20260620 11:42:48.001302 12160 consensus_queue.cc:260] T 0bfa55ce3788466c87a0fa6571939ff1 P 84b9816672b94db488d7a779f4e2b8e6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7ada3c0dc1074859bdb2dcc9a582a460" member_type: VOTER last_known_addr { host: "127.6.234.193" port: 38049 } } peers { permanent_uuid: "84b9816672b94db488d7a779f4e2b8e6" member_type: VOTER last_known_addr { host: "127.6.234.194" port: 33253 } } peers { permanent_uuid: "a2e137871e0648f4aaf774ff62871920" member_type: VOTER last_known_addr { host: "127.6.234.195" port: 42715 } }
I20260620 11:42:48.001610 12159 ts_tablet_manager.cc:1428] T 0bfa55ce3788466c87a0fa6571939ff1 P 7ada3c0dc1074859bdb2dcc9a582a460: Time spent starting tablet: real 0.003s user 0.003s sys 0.000s
I20260620 11:42:48.001673 12160 ts_tablet_manager.cc:1428] T 0bfa55ce3788466c87a0fa6571939ff1 P 84b9816672b94db488d7a779f4e2b8e6: Time spent starting tablet: real 0.003s user 0.003s sys 0.000s
I20260620 11:42:48.001788 11883 heartbeater.cc:503] Master 127.6.234.254:45257 was elected leader, sending a full tablet report...
I20260620 11:42:48.001834 12014 heartbeater.cc:503] Master 127.6.234.254:45257 was elected leader, sending a full tablet report...
I20260620 11:42:48.002290 12161 raft_consensus.cc:348] T 0bfa55ce3788466c87a0fa6571939ff1 P a2e137871e0648f4aaf774ff62871920 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7ada3c0dc1074859bdb2dcc9a582a460" member_type: VOTER last_known_addr { host: "127.6.234.193" port: 38049 } } peers { permanent_uuid: "84b9816672b94db488d7a779f4e2b8e6" member_type: VOTER last_known_addr { host: "127.6.234.194" port: 33253 } } peers { permanent_uuid: "a2e137871e0648f4aaf774ff62871920" member_type: VOTER last_known_addr { host: "127.6.234.195" port: 42715 } }
I20260620 11:42:48.002446 12161 raft_consensus.cc:374] T 0bfa55ce3788466c87a0fa6571939ff1 P a2e137871e0648f4aaf774ff62871920 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 11:42:48.002485 12161 raft_consensus.cc:729] T 0bfa55ce3788466c87a0fa6571939ff1 P a2e137871e0648f4aaf774ff62871920 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a2e137871e0648f4aaf774ff62871920, State: Initialized, Role: FOLLOWER
I20260620 11:42:48.002586 12161 consensus_queue.cc:260] T 0bfa55ce3788466c87a0fa6571939ff1 P a2e137871e0648f4aaf774ff62871920 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7ada3c0dc1074859bdb2dcc9a582a460" member_type: VOTER last_known_addr { host: "127.6.234.193" port: 38049 } } peers { permanent_uuid: "84b9816672b94db488d7a779f4e2b8e6" member_type: VOTER last_known_addr { host: "127.6.234.194" port: 33253 } } peers { permanent_uuid: "a2e137871e0648f4aaf774ff62871920" member_type: VOTER last_known_addr { host: "127.6.234.195" port: 42715 } }
I20260620 11:42:48.002871 12145 heartbeater.cc:503] Master 127.6.234.254:45257 was elected leader, sending a full tablet report...
I20260620 11:42:48.003096 12161 ts_tablet_manager.cc:1428] T 0bfa55ce3788466c87a0fa6571939ff1 P a2e137871e0648f4aaf774ff62871920: Time spent starting tablet: real 0.004s user 0.004s sys 0.000s
W20260620 11:42:48.106822 12015 tablet.cc:2369] T 0bfa55ce3788466c87a0fa6571939ff1 P 84b9816672b94db488d7a779f4e2b8e6: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260620 11:42:48.157889 12165 raft_consensus.cc:482] T 0bfa55ce3788466c87a0fa6571939ff1 P 7ada3c0dc1074859bdb2dcc9a582a460 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260620 11:42:48.158007 12165 raft_consensus.cc:504] T 0bfa55ce3788466c87a0fa6571939ff1 P 7ada3c0dc1074859bdb2dcc9a582a460 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7ada3c0dc1074859bdb2dcc9a582a460" member_type: VOTER last_known_addr { host: "127.6.234.193" port: 38049 } } peers { permanent_uuid: "84b9816672b94db488d7a779f4e2b8e6" member_type: VOTER last_known_addr { host: "127.6.234.194" port: 33253 } } peers { permanent_uuid: "a2e137871e0648f4aaf774ff62871920" member_type: VOTER last_known_addr { host: "127.6.234.195" port: 42715 } }
I20260620 11:42:48.158424 12165 leader_election.cc:288] T 0bfa55ce3788466c87a0fa6571939ff1 P 7ada3c0dc1074859bdb2dcc9a582a460 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 84b9816672b94db488d7a779f4e2b8e6 (127.6.234.194:33253), a2e137871e0648f4aaf774ff62871920 (127.6.234.195:42715)
I20260620 11:42:48.163470 12157 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 11:42:48.158766 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.6.234.193:58873 (local address 127.6.234.195:42715)
0620 11:42:48.158828 (+ 62us) server_negotiation.cc:206] Beginning negotiation
0620 11:42:48.158831 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0620 11:42:48.158839 (+ 8us) server_negotiation.cc:407] Connection header received
0620 11:42:48.158875 (+ 36us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 11:42:48.158875 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 11:42:48.158879 (+ 4us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 11:42:48.158951 (+ 72us) server_negotiation.cc:226] Negotiated authn=SASL
0620 11:42:48.163196 (+ 4245us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 11:42:48.163198 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 11:42:48.163198 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 11:42:48.163204 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0620 11:42:48.163222 (+ 18us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 11:42:48.163227 (+ 5us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 11:42:48.163293 (+ 66us) server_negotiation.cc:1035] Waiting for connection context
0620 11:42:48.163380 (+ 87us) server_negotiation.cc:299] Negotiation successful
0620 11:42:48.163397 (+ 17us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":40}
I20260620 11:42:48.163470 12158 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 11:42:48.158780 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.6.234.193:33863 (local address 127.6.234.194:33253)
0620 11:42:48.158830 (+ 50us) server_negotiation.cc:206] Beginning negotiation
0620 11:42:48.158832 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 11:42:48.158915 (+ 83us) server_negotiation.cc:407] Connection header received
0620 11:42:48.158945 (+ 30us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 11:42:48.158946 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 11:42:48.158951 (+ 5us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 11:42:48.159016 (+ 65us) server_negotiation.cc:226] Negotiated authn=SASL
0620 11:42:48.163196 (+ 4180us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 11:42:48.163198 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 11:42:48.163198 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 11:42:48.163204 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0620 11:42:48.163222 (+ 18us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 11:42:48.163228 (+ 6us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 11:42:48.163293 (+ 65us) server_negotiation.cc:1035] Waiting for connection context
0620 11:42:48.163380 (+ 87us) server_negotiation.cc:299] Negotiation successful
0620 11:42:48.163397 (+ 17us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":23}
I20260620 11:42:48.163472 11885 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 11:42:48.158621 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.234.195:42715 (local address 127.6.234.193:58873)
0620 11:42:48.158678 (+ 57us) negotiation.cc:107] Waiting for socket to connect
0620 11:42:48.158690 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0620 11:42:48.158751 (+ 61us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 11:42:48.159008 (+ 257us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 11:42:48.159009 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 11:42:48.159012 (+ 3us) client_negotiation.cc:189] Negotiated authn=SASL
0620 11:42:48.163061 (+ 4049us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0620 11:42:48.163062 (+ 1us) client_negotiation.cc:656] calling sasl_client_start()
0620 11:42:48.163071 (+ 9us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0620 11:42:48.163087 (+ 16us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 11:42:48.163302 (+ 215us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 11:42:48.163303 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 11:42:48.163303 (+ 0us) client_negotiation.cc:769] Sending connection context
0620 11:42:48.163357 (+ 54us) client_negotiation.cc:240] Negotiation successful
0620 11:42:48.163373 (+ 16us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":35}
I20260620 11:42:48.163486 12168 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 11:42:48.158644 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.234.194:33253 (local address 127.6.234.193:33863)
0620 11:42:48.158836 (+ 192us) negotiation.cc:107] Waiting for socket to connect
0620 11:42:48.158854 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0620 11:42:48.158904 (+ 50us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 11:42:48.159031 (+ 127us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 11:42:48.159032 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 11:42:48.159034 (+ 2us) client_negotiation.cc:189] Negotiated authn=SASL
0620 11:42:48.163061 (+ 4027us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0620 11:42:48.163062 (+ 1us) client_negotiation.cc:656] calling sasl_client_start()
0620 11:42:48.163071 (+ 9us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0620 11:42:48.163087 (+ 16us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 11:42:48.163302 (+ 215us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 11:42:48.163303 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 11:42:48.163303 (+ 0us) client_negotiation.cc:769] Sending connection context
0620 11:42:48.163369 (+ 66us) client_negotiation.cc:240] Negotiation successful
0620 11:42:48.163379 (+ 10us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":173,"mutex_wait_us":3,"thread_start_us":114,"threads_started":1}
I20260620 11:42:48.163825 11968 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0bfa55ce3788466c87a0fa6571939ff1" candidate_uuid: "7ada3c0dc1074859bdb2dcc9a582a460" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "84b9816672b94db488d7a779f4e2b8e6" is_pre_election: true
I20260620 11:42:48.163830 12099 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0bfa55ce3788466c87a0fa6571939ff1" candidate_uuid: "7ada3c0dc1074859bdb2dcc9a582a460" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a2e137871e0648f4aaf774ff62871920" is_pre_election: true
I20260620 11:42:48.163991 12099 raft_consensus.cc:2454] T 0bfa55ce3788466c87a0fa6571939ff1 P a2e137871e0648f4aaf774ff62871920 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 7ada3c0dc1074859bdb2dcc9a582a460 in term 0.
I20260620 11:42:48.163992 11968 raft_consensus.cc:2454] T 0bfa55ce3788466c87a0fa6571939ff1 P 84b9816672b94db488d7a779f4e2b8e6 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 7ada3c0dc1074859bdb2dcc9a582a460 in term 0.
I20260620 11:42:48.164273 11773 leader_election.cc:302] T 0bfa55ce3788466c87a0fa6571939ff1 P 7ada3c0dc1074859bdb2dcc9a582a460 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 7ada3c0dc1074859bdb2dcc9a582a460, a2e137871e0648f4aaf774ff62871920; no voters:
I20260620 11:42:48.164409 12165 raft_consensus.cc:2781] T 0bfa55ce3788466c87a0fa6571939ff1 P 7ada3c0dc1074859bdb2dcc9a582a460 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260620 11:42:48.164460 12165 raft_consensus.cc:482] T 0bfa55ce3788466c87a0fa6571939ff1 P 7ada3c0dc1074859bdb2dcc9a582a460 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260620 11:42:48.164490 12165 raft_consensus.cc:3037] T 0bfa55ce3788466c87a0fa6571939ff1 P 7ada3c0dc1074859bdb2dcc9a582a460 [term 0 FOLLOWER]: Advancing to term 1
I20260620 11:42:48.165249 12165 raft_consensus.cc:504] T 0bfa55ce3788466c87a0fa6571939ff1 P 7ada3c0dc1074859bdb2dcc9a582a460 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7ada3c0dc1074859bdb2dcc9a582a460" member_type: VOTER last_known_addr { host: "127.6.234.193" port: 38049 } } peers { permanent_uuid: "84b9816672b94db488d7a779f4e2b8e6" member_type: VOTER last_known_addr { host: "127.6.234.194" port: 33253 } } peers { permanent_uuid: "a2e137871e0648f4aaf774ff62871920" member_type: VOTER last_known_addr { host: "127.6.234.195" port: 42715 } }
I20260620 11:42:48.165405 12165 leader_election.cc:288] T 0bfa55ce3788466c87a0fa6571939ff1 P 7ada3c0dc1074859bdb2dcc9a582a460 [CANDIDATE]: Term 1 election: Requested vote from peers 84b9816672b94db488d7a779f4e2b8e6 (127.6.234.194:33253), a2e137871e0648f4aaf774ff62871920 (127.6.234.195:42715)
I20260620 11:42:48.165560 11968 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0bfa55ce3788466c87a0fa6571939ff1" candidate_uuid: "7ada3c0dc1074859bdb2dcc9a582a460" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "84b9816672b94db488d7a779f4e2b8e6"
I20260620 11:42:48.165578 12099 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0bfa55ce3788466c87a0fa6571939ff1" candidate_uuid: "7ada3c0dc1074859bdb2dcc9a582a460" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a2e137871e0648f4aaf774ff62871920"
I20260620 11:42:48.165630 11968 raft_consensus.cc:3037] T 0bfa55ce3788466c87a0fa6571939ff1 P 84b9816672b94db488d7a779f4e2b8e6 [term 0 FOLLOWER]: Advancing to term 1
I20260620 11:42:48.165664 12099 raft_consensus.cc:3037] T 0bfa55ce3788466c87a0fa6571939ff1 P a2e137871e0648f4aaf774ff62871920 [term 0 FOLLOWER]: Advancing to term 1
I20260620 11:42:48.166324 11968 raft_consensus.cc:2454] T 0bfa55ce3788466c87a0fa6571939ff1 P 84b9816672b94db488d7a779f4e2b8e6 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 7ada3c0dc1074859bdb2dcc9a582a460 in term 1.
I20260620 11:42:48.166455 12099 raft_consensus.cc:2454] T 0bfa55ce3788466c87a0fa6571939ff1 P a2e137871e0648f4aaf774ff62871920 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 7ada3c0dc1074859bdb2dcc9a582a460 in term 1.
I20260620 11:42:48.166510 11771 leader_election.cc:302] T 0bfa55ce3788466c87a0fa6571939ff1 P 7ada3c0dc1074859bdb2dcc9a582a460 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 7ada3c0dc1074859bdb2dcc9a582a460, 84b9816672b94db488d7a779f4e2b8e6; no voters:
I20260620 11:42:48.166599 12165 raft_consensus.cc:2781] T 0bfa55ce3788466c87a0fa6571939ff1 P 7ada3c0dc1074859bdb2dcc9a582a460 [term 1 FOLLOWER]: Leader election won for term 1
I20260620 11:42:48.166734 12165 raft_consensus.cc:686] T 0bfa55ce3788466c87a0fa6571939ff1 P 7ada3c0dc1074859bdb2dcc9a582a460 [term 1 LEADER]: Becoming Leader. State: Replica: 7ada3c0dc1074859bdb2dcc9a582a460, State: Running, Role: LEADER
I20260620 11:42:48.166834 12165 consensus_queue.cc:237] T 0bfa55ce3788466c87a0fa6571939ff1 P 7ada3c0dc1074859bdb2dcc9a582a460 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7ada3c0dc1074859bdb2dcc9a582a460" member_type: VOTER last_known_addr { host: "127.6.234.193" port: 38049 } } peers { permanent_uuid: "84b9816672b94db488d7a779f4e2b8e6" member_type: VOTER last_known_addr { host: "127.6.234.194" port: 33253 } } peers { permanent_uuid: "a2e137871e0648f4aaf774ff62871920" member_type: VOTER last_known_addr { host: "127.6.234.195" port: 42715 } }
I20260620 11:42:48.167738 11695 catalog_manager.cc:5495] T 0bfa55ce3788466c87a0fa6571939ff1 P 7ada3c0dc1074859bdb2dcc9a582a460 reported cstate change: term changed from 0 to 1, leader changed from <none> to 7ada3c0dc1074859bdb2dcc9a582a460 (127.6.234.193). New cstate: current_term: 1 leader_uuid: "7ada3c0dc1074859bdb2dcc9a582a460" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7ada3c0dc1074859bdb2dcc9a582a460" member_type: VOTER last_known_addr { host: "127.6.234.193" port: 38049 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "84b9816672b94db488d7a779f4e2b8e6" member_type: VOTER last_known_addr { host: "127.6.234.194" port: 33253 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "a2e137871e0648f4aaf774ff62871920" member_type: VOTER last_known_addr { host: "127.6.234.195" port: 42715 } health_report { overall_health: UNKNOWN } } }
I20260620 11:42:48.172295 12155 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 11:42:48.171256 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54592 (local address 127.6.234.193:38049)
0620 11:42:48.171315 (+ 59us) server_negotiation.cc:206] Beginning negotiation
0620 11:42:48.171317 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 11:42:48.171326 (+ 9us) server_negotiation.cc:407] Connection header received
0620 11:42:48.171398 (+ 72us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 11:42:48.171399 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 11:42:48.171412 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 11:42:48.171462 (+ 50us) server_negotiation.cc:226] Negotiated authn=SASL
0620 11:42:48.172114 (+ 652us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 11:42:48.172115 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 11:42:48.172115 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 11:42:48.172119 (+ 4us) server_negotiation.cc:922] calling sasl_server_start()
0620 11:42:48.172135 (+ 16us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 11:42:48.172138 (+ 3us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 11:42:48.172189 (+ 51us) server_negotiation.cc:1035] Waiting for connection context
0620 11:42:48.172236 (+ 47us) server_negotiation.cc:299] Negotiation successful
0620 11:42:48.172247 (+ 11us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":37}
I20260620 11:42:48.173887 12099 raft_consensus.cc:1261] T 0bfa55ce3788466c87a0fa6571939ff1 P a2e137871e0648f4aaf774ff62871920 [term 1 FOLLOWER]: Refusing update from remote peer 7ada3c0dc1074859bdb2dcc9a582a460: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 11:42:48.173885 11968 raft_consensus.cc:1261] T 0bfa55ce3788466c87a0fa6571939ff1 P 84b9816672b94db488d7a779f4e2b8e6 [term 1 FOLLOWER]: Refusing update from remote peer 7ada3c0dc1074859bdb2dcc9a582a460: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 11:42:48.174201 12165 consensus_queue.cc:1035] T 0bfa55ce3788466c87a0fa6571939ff1 P 7ada3c0dc1074859bdb2dcc9a582a460 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a2e137871e0648f4aaf774ff62871920" member_type: VOTER last_known_addr { host: "127.6.234.195" port: 42715 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 11:42:48.174362 12169 consensus_queue.cc:1035] T 0bfa55ce3788466c87a0fa6571939ff1 P 7ada3c0dc1074859bdb2dcc9a582a460 [LEADER]: Connected to new peer: Peer: permanent_uuid: "84b9816672b94db488d7a779f4e2b8e6" member_type: VOTER last_known_addr { host: "127.6.234.194" port: 33253 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 11:42:48.176770 12173 mvcc.cc:205] Tried to move back new op lower bound from 7298890826437804032 to 7298890826411999232. Current Snapshot: MvccSnapshot[applied={T|T < 7298890826437804032}]
I20260620 11:42:48.177286 12172 mvcc.cc:205] Tried to move back new op lower bound from 7298890826437804032 to 7298890826411999232. Current Snapshot: MvccSnapshot[applied={T|T < 7298890826437804032 or (T in {7298890826437804032})}]
I20260620 11:42:48.178483 12175 mvcc.cc:205] Tried to move back new op lower bound from 7298890826437804032 to 7298890826411999232. Current Snapshot: MvccSnapshot[applied={T|T < 7298890826437804032}]
I20260620 11:42:48.178787 11694 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:50278:
table { table_name: "test-table" } modify_external_catalogs: true
I20260620 11:42:48.178879 11694 catalog_manager.cc:2645] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:50278:
table { table_name: "test-table" } modify_external_catalogs: true
I20260620 11:42:48.179574 11694 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 0a05dc3731924f579e3faec7c677c80c: Sending DeleteTablet for 3 replicas of tablet 0bfa55ce3788466c87a0fa6571939ff1
I20260620 11:42:48.179912 11817 tablet_service.cc:1514] Processing DeleteTablet for tablet 0bfa55ce3788466c87a0fa6571939ff1 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 11:42:48 UTC) from {username='slave'} at 127.0.0.1:54580
I20260620 11:42:48.179900 12079 tablet_service.cc:1514] Processing DeleteTablet for tablet 0bfa55ce3788466c87a0fa6571939ff1 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 11:42:48 UTC) from {username='slave'} at 127.0.0.1:37058
I20260620 11:42:48.179998 11948 tablet_service.cc:1514] Processing DeleteTablet for tablet 0bfa55ce3788466c87a0fa6571939ff1 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 11:42:48 UTC) from {username='slave'} at 127.0.0.1:44572
I20260620 11:42:48.180241 12184 tablet_replica.cc:330] T 0bfa55ce3788466c87a0fa6571939ff1 P 84b9816672b94db488d7a779f4e2b8e6: stopping tablet replica
I20260620 11:42:48.180279 7083 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task1JRg7N/build/release/bin/kudu with pid 11755
I20260620 11:42:48.180352 12184 raft_consensus.cc:2229] T 0bfa55ce3788466c87a0fa6571939ff1 P 84b9816672b94db488d7a779f4e2b8e6 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260620 11:42:48.180450 12184 raft_consensus.cc:2258] T 0bfa55ce3788466c87a0fa6571939ff1 P 84b9816672b94db488d7a779f4e2b8e6 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 11:42:48.180795 12183 tablet_replica.cc:330] T 0bfa55ce3788466c87a0fa6571939ff1 P a2e137871e0648f4aaf774ff62871920: stopping tablet replica
I20260620 11:42:48.180902 12183 raft_consensus.cc:2229] T 0bfa55ce3788466c87a0fa6571939ff1 P a2e137871e0648f4aaf774ff62871920 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260620 11:42:48.181000 12183 raft_consensus.cc:2258] T 0bfa55ce3788466c87a0fa6571939ff1 P a2e137871e0648f4aaf774ff62871920 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 11:42:48.181304 12184 ts_tablet_manager.cc:1905] T 0bfa55ce3788466c87a0fa6571939ff1 P 84b9816672b94db488d7a779f4e2b8e6: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 11:42:48.181463 12183 ts_tablet_manager.cc:1905] T 0bfa55ce3788466c87a0fa6571939ff1 P a2e137871e0648f4aaf774ff62871920: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 11:42:48.182972 12184 ts_tablet_manager.cc:1918] T 0bfa55ce3788466c87a0fa6571939ff1 P 84b9816672b94db488d7a779f4e2b8e6: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260620 11:42:48.183054 12184 log.cc:1196] T 0bfa55ce3788466c87a0fa6571939ff1 P 84b9816672b94db488d7a779f4e2b8e6: Deleting WAL directory at /tmp/dist-test-task1JRg7N/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781955746769051-7083-0/minicluster-data/ts-1/wal/wals/0bfa55ce3788466c87a0fa6571939ff1
I20260620 11:42:48.183317 12184 ts_tablet_manager.cc:1939] T 0bfa55ce3788466c87a0fa6571939ff1 P 84b9816672b94db488d7a779f4e2b8e6: Deleting consensus metadata
I20260620 11:42:48.183385 12183 ts_tablet_manager.cc:1918] T 0bfa55ce3788466c87a0fa6571939ff1 P a2e137871e0648f4aaf774ff62871920: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260620 11:42:48.183459 12183 log.cc:1196] T 0bfa55ce3788466c87a0fa6571939ff1 P a2e137871e0648f4aaf774ff62871920: Deleting WAL directory at /tmp/dist-test-task1JRg7N/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781955746769051-7083-0/minicluster-data/ts-2/wal/wals/0bfa55ce3788466c87a0fa6571939ff1
I20260620 11:42:48.183709 12183 ts_tablet_manager.cc:1939] T 0bfa55ce3788466c87a0fa6571939ff1 P a2e137871e0648f4aaf774ff62871920: Deleting consensus metadata
I20260620 11:42:48.183730 11680 catalog_manager.cc:4841] TS 84b9816672b94db488d7a779f4e2b8e6 (127.6.234.194:33253): tablet 0bfa55ce3788466c87a0fa6571939ff1 (table test-table [id=a0cda2612ce5495eb218c8fd7dd48b04]) successfully deleted
I20260620 11:42:48.184073 11682 catalog_manager.cc:4841] TS a2e137871e0648f4aaf774ff62871920 (127.6.234.195:42715): tablet 0bfa55ce3788466c87a0fa6571939ff1 (table test-table [id=a0cda2612ce5495eb218c8fd7dd48b04]) successfully deleted
I20260620 11:42:48.187275 7083 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task1JRg7N/build/release/bin/kudu with pid 11886
W20260620 11:42:48.187290 11680 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.6.234.193:38049 (error 108)
I20260620 11:42:48.188777 12154 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 11:42:48.188632 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.234.193:38049 (local address 127.0.0.1:54606)
0620 11:42:48.188687 (+ 55us) negotiation.cc:107] Waiting for socket to connect
0620 11:42:48.188728 (+ 41us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.234.193:38049: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":35}
W20260620 11:42:48.188918 11680 catalog_manager.cc:4584] TS 7ada3c0dc1074859bdb2dcc9a582a460 (127.6.234.193:38049): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 0bfa55ce3788466c87a0fa6571939ff1: Network error: Client connection negotiation failed: client connection to 127.6.234.193:38049: connect: Connection refused (error 111)
I20260620 11:42:48.191951 7083 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task1JRg7N/build/release/bin/kudu with pid 12017
I20260620 11:42:48.196841 7083 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task1JRg7N/build/release/bin/kudu with pid 11664
2026-06-20T11:42:48Z chronyd exiting
[ OK ] SecurityITest.TestDisableAuthenticationEncryption (733 ms)
[ RUN ] SecurityITest.TestJwtMiniCluster
I20260620 11:42:48.215819 7083 mini_oidc.cc:152] Starting JWKS server
I20260620 11:42:48.216837 7083 webserver.cc:460] Webserver started at https://127.0.0.1:43275/ using document root <none> and password file <none>
I20260620 11:42:48.216960 7083 mini_oidc.cc:189] Starting OIDC Discovery server
I20260620 11:42:48.217136 7083 webserver.cc:460] Webserver started at http://127.0.0.1:46853/ using document root <none> and password file <none>
2026-06-20T11:42:48Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-20T11:42:48Z Disabled control of system clock
I20260620 11:42:48.224637 7083 external_mini_cluster.cc:1326] Running /tmp/dist-test-task1JRg7N/build/release/bin/kudu
/tmp/dist-test-task1JRg7N/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task1JRg7N/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1781955746769051-7083-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task1JRg7N/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1781955746769051-7083-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task1JRg7N/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1781955746769051-7083-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task1JRg7N/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1781955746769051-7083-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.6.234.254:36633
--webserver_interface=127.6.234.254
--webserver_port=0
--builtin_ntp_servers=127.6.234.212:44493
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.6.234.254:36633
--enable_jwt_token_auth=true
--jwks_url=https://localhost:43275/jwks.json
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--trusted_certificate_file=/tmp/dist-test-task1JRg7N/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1781955746769051-7083-0/testchainca.cert with env {}
W20260620 11:42:48.302997 12191 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 11:42:48.303193 12191 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 11:42:48.303215 12191 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 11:42:48.304388 12191 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 11:42:48.304430 12191 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 11:42:48.304443 12191 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 11:42:48.304456 12191 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 11:42:48.304468 12191 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
W20260620 11:42:48.304481 12191 flags.cc:425] Enabled experimental flag: --enable_jwt_token_auth=true
W20260620 11:42:48.304492 12191 flags.cc:425] Enabled experimental flag: --jwks_url=https://localhost:43275/jwks.json
I20260620 11:42:48.306488 12191 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.6.234.212:44493
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task1JRg7N/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1781955746769051-7083-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task1JRg7N/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1781955746769051-7083-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.6.234.254:36633
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.6.234.254:36633
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--enable_jwt_token_auth=true
--jwks_url=https://localhost:43275/jwks.json
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task1JRg7N/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1781955746769051-7083-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.6.234.254
--webserver_port=0
--trusted_certificate_file=/tmp/dist-test-task1JRg7N/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1781955746769051-7083-0/testchainca.cert
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.12191
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task1JRg7N/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1781955746769051-7083-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 20 Jun 2026 10:32:37 UTC on ccycloud.quasar-bgyofv.root.comops.site
build id 4457173
I20260620 11:42:48.306680 12191 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 11:42:48.306841 12191 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 11:42:48.311646 12197 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 11:42:48.311626 12196 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 11:42:48.311656 12191 server_base.cc:1034] running on GCE node
W20260620 11:42:48.311638 12199 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 11:42:48.312122 12191 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 11:42:48.312368 12191 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 11:42:48.313515 12191 hybrid_clock.cc:648] HybridClock initialized: now 1781955768313491 us; error 38 us; skew 500 ppm
I20260620 11:42:48.314783 12191 webserver.cc:460] Webserver started at http://127.6.234.254:43091/ using document root <none> and password file <none>
I20260620 11:42:48.314978 12191 fs_manager.cc:359] Metadata directory not provided
I20260620 11:42:48.315017 12191 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 11:42:48.315100 12191 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 11:42:48.316064 12191 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task1JRg7N/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1781955746769051-7083-0/minicluster-data/master-0/data/instance:
uuid: "9dfb0ab9e94343858e319c7612a75f13"
format_stamp: "Formatted at 2026-06-20 11:42:48 on dist-test-slave-cjc6"
server_key: "62a4070ac23d3ef66f7eb54bf554f0b0"
server_key_iv: "261915b6506958c4075dd37ae98488da"
server_key_version: "encryptionkey@0"
I20260620 11:42:48.316391 12191 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task1JRg7N/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1781955746769051-7083-0/minicluster-data/master-0/wal/instance:
uuid: "9dfb0ab9e94343858e319c7612a75f13"
format_stamp: "Formatted at 2026-06-20 11:42:48 on dist-test-slave-cjc6"
server_key: "62a4070ac23d3ef66f7eb54bf554f0b0"
server_key_iv: "261915b6506958c4075dd37ae98488da"
server_key_version: "encryptionkey@0"
I20260620 11:42:48.317512 12191 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.003s
I20260620 11:42:48.318209 12205 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 11:42:48.318439 12191 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.000s
I20260620 11:42:48.318539 12191 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task1JRg7N/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1781955746769051-7083-0/minicluster-data/master-0/data,/tmp/dist-test-task1JRg7N/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1781955746769051-7083-0/minicluster-data/master-0/wal
uuid: "9dfb0ab9e94343858e319c7612a75f13"
format_stamp: "Formatted at 2026-06-20 11:42:48 on dist-test-slave-cjc6"
server_key: "62a4070ac23d3ef66f7eb54bf554f0b0"
server_key_iv: "261915b6506958c4075dd37ae98488da"
server_key_version: "encryptionkey@0"
I20260620 11:42:48.318629 12191 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task1JRg7N/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1781955746769051-7083-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task1JRg7N/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1781955746769051-7083-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task1JRg7N/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1781955746769051-7083-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 11:42:48.340131 12191 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 11:42:48.340459 12191 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 11:42:48.340615 12191 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 11:42:48.344573 12191 rpc_server.cc:307] RPC server started. Bound to: 127.6.234.254:36633
I20260620 11:42:48.344606 12257 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.6.234.254:36633 every 8 connection(s)
I20260620 11:42:48.344954 12191 server_base.cc:1166] Dumped server information to /tmp/dist-test-task1JRg7N/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1781955746769051-7083-0/minicluster-data/master-0/data/info.pb
I20260620 11:42:48.345988 12258 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 11:42:48.348213 12258 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 9dfb0ab9e94343858e319c7612a75f13: Bootstrap starting.
I20260620 11:42:48.348767 12258 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 9dfb0ab9e94343858e319c7612a75f13: Neither blocks nor log segments found. Creating new log.
I20260620 11:42:48.348769 7083 external_mini_cluster.cc:1388] Started /tmp/dist-test-task1JRg7N/build/release/bin/kudu as pid 12191
I20260620 11:42:48.348902 7083 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task1JRg7N/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1781955746769051-7083-0/minicluster-data/master-0/wal/instance
I20260620 11:42:48.349081 7083 external_mini_cluster.cc:1428] Setting key 488e2d20e81714dc45549f61df7eda9a
I20260620 11:42:48.349097 12258 log.cc:824] T 00000000000000000000000000000000 P 9dfb0ab9e94343858e319c7612a75f13: Log is configured to *not* fsync() on all Append() calls
I20260620 11:42:48.349874 12258 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 9dfb0ab9e94343858e319c7612a75f13: No bootstrap required, opened a new log
I20260620 11:42:48.351876 12258 raft_consensus.cc:348] T 00000000000000000000000000000000 P 9dfb0ab9e94343858e319c7612a75f13 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9dfb0ab9e94343858e319c7612a75f13" member_type: VOTER last_known_addr { host: "127.6.234.254" port: 36633 } }
I20260620 11:42:48.352030 12258 raft_consensus.cc:374] T 00000000000000000000000000000000 P 9dfb0ab9e94343858e319c7612a75f13 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 11:42:48.352080 12258 raft_consensus.cc:729] T 00000000000000000000000000000000 P 9dfb0ab9e94343858e319c7612a75f13 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9dfb0ab9e94343858e319c7612a75f13, State: Initialized, Role: FOLLOWER
I20260620 11:42:48.352201 12258 consensus_queue.cc:260] T 00000000000000000000000000000000 P 9dfb0ab9e94343858e319c7612a75f13 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9dfb0ab9e94343858e319c7612a75f13" member_type: VOTER last_known_addr { host: "127.6.234.254" port: 36633 } }
I20260620 11:42:48.352272 12258 raft_consensus.cc:388] T 00000000000000000000000000000000 P 9dfb0ab9e94343858e319c7612a75f13 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260620 11:42:48.352295 12258 raft_consensus.cc:482] T 00000000000000000000000000000000 P 9dfb0ab9e94343858e319c7612a75f13 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260620 11:42:48.352365 12258 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 9dfb0ab9e94343858e319c7612a75f13 [term 0 FOLLOWER]: Advancing to term 1
I20260620 11:42:48.353011 12258 raft_consensus.cc:504] T 00000000000000000000000000000000 P 9dfb0ab9e94343858e319c7612a75f13 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9dfb0ab9e94343858e319c7612a75f13" member_type: VOTER last_known_addr { host: "127.6.234.254" port: 36633 } }
I20260620 11:42:48.353134 12258 leader_election.cc:302] T 00000000000000000000000000000000 P 9dfb0ab9e94343858e319c7612a75f13 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 9dfb0ab9e94343858e319c7612a75f13; no voters:
I20260620 11:42:48.353289 12258 leader_election.cc:288] T 00000000000000000000000000000000 P 9dfb0ab9e94343858e319c7612a75f13 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260620 11:42:48.353313 12263 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 9dfb0ab9e94343858e319c7612a75f13 [term 1 FOLLOWER]: Leader election won for term 1
I20260620 11:42:48.353422 12263 raft_consensus.cc:686] T 00000000000000000000000000000000 P 9dfb0ab9e94343858e319c7612a75f13 [term 1 LEADER]: Becoming Leader. State: Replica: 9dfb0ab9e94343858e319c7612a75f13, State: Running, Role: LEADER
I20260620 11:42:48.353503 12263 consensus_queue.cc:237] T 00000000000000000000000000000000 P 9dfb0ab9e94343858e319c7612a75f13 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9dfb0ab9e94343858e319c7612a75f13" member_type: VOTER last_known_addr { host: "127.6.234.254" port: 36633 } }
I20260620 11:42:48.353565 12258 sys_catalog.cc:564] T 00000000000000000000000000000000 P 9dfb0ab9e94343858e319c7612a75f13 [sys.catalog]: configured and running, proceeding with master startup.
I20260620 11:42:48.353850 12265 sys_catalog.cc:455] T 00000000000000000000000000000000 P 9dfb0ab9e94343858e319c7612a75f13 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 9dfb0ab9e94343858e319c7612a75f13. Latest consensus state: current_term: 1 leader_uuid: "9dfb0ab9e94343858e319c7612a75f13" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9dfb0ab9e94343858e319c7612a75f13" member_type: VOTER last_known_addr { host: "127.6.234.254" port: 36633 } } }
I20260620 11:42:48.353943 12265 sys_catalog.cc:458] T 00000000000000000000000000000000 P 9dfb0ab9e94343858e319c7612a75f13 [sys.catalog]: This master's current role is: LEADER
I20260620 11:42:48.354120 12264 sys_catalog.cc:455] T 00000000000000000000000000000000 P 9dfb0ab9e94343858e319c7612a75f13 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "9dfb0ab9e94343858e319c7612a75f13" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9dfb0ab9e94343858e319c7612a75f13" member_type: VOTER last_known_addr { host: "127.6.234.254" port: 36633 } } }
I20260620 11:42:48.354194 12264 sys_catalog.cc:458] T 00000000000000000000000000000000 P 9dfb0ab9e94343858e319c7612a75f13 [sys.catalog]: This master's current role is: LEADER
I20260620 11:42:48.354183 12268 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260620 11:42:48.354730 12262 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 11:42:48.349627 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48450 (local address 127.6.234.254:36633)
0620 11:42:48.349831 (+ 204us) server_negotiation.cc:206] Beginning negotiation
0620 11:42:48.349841 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0620 11:42:48.349873 (+ 32us) server_negotiation.cc:407] Connection header received
0620 11:42:48.350339 (+ 466us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 11:42:48.350347 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 11:42:48.350405 (+ 58us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 11:42:48.350553 (+ 148us) server_negotiation.cc:226] Negotiated authn=SASL
0620 11:42:48.350769 (+ 216us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 11:42:48.351606 (+ 837us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 11:42:48.352602 (+ 996us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 11:42:48.353002 (+ 400us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 11:42:48.353132 (+ 130us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 11:42:48.353828 (+ 696us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 11:42:48.353832 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 11:42:48.353833 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 11:42:48.354043 (+ 210us) server_negotiation.cc:922] calling sasl_server_start()
0620 11:42:48.354088 (+ 45us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 11:42:48.354104 (+ 16us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 11:42:48.354165 (+ 61us) server_negotiation.cc:1035] Waiting for connection context
0620 11:42:48.354459 (+ 294us) server_negotiation.cc:299] Negotiation successful
0620 11:42:48.354535 (+ 76us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":122,"thread_start_us":94,"threads_started":1}
I20260620 11:42:48.355350 12268 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260620 11:42:48.355552 7083 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260620 11:42:48.356959 12268 catalog_manager.cc:1347] Generated new cluster ID: cd2743524b38492e9da15674a7a0d317
I20260620 11:42:48.357012 12268 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260620 11:42:48.376255 12268 catalog_manager.cc:1370] Generated new certificate authority record
I20260620 11:42:48.376838 12268 catalog_manager.cc:1504] Loading token signing keys...
I20260620 11:42:48.387960 12268 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 9dfb0ab9e94343858e319c7612a75f13: Generated new TSK 0
I20260620 11:42:48.388213 12268 catalog_manager.cc:1514] Initializing in-progress tserver states...
E20260620 11:42:48.416428 12288 webserver.cc:533] Webserver: sslize failed: error:14094415:SSL routines:ssl3_read_bytes:sslv3 alert certificate expired
E20260620 11:42:48.416536 12288 webserver.cc:533] Webserver: error shutting down SSL connection: error:140E0197:SSL routines:SSL_shutdown:shutdown while in init
I20260620 11:42:48.416816 12262 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 11:42:48.401087 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48458 (local address 127.6.234.254:36633)
0620 11:42:48.401135 (+ 48us) server_negotiation.cc:206] Beginning negotiation
0620 11:42:48.401137 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 11:42:48.401271 (+ 134us) server_negotiation.cc:407] Connection header received
0620 11:42:48.401352 (+ 81us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 11:42:48.401353 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 11:42:48.401366 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 11:42:48.401445 (+ 79us) server_negotiation.cc:226] Negotiated authn=JWT
0620 11:42:48.401588 (+ 143us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 11:42:48.402395 (+ 807us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 11:42:48.403236 (+ 841us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 11:42:48.403580 (+ 344us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 11:42:48.403652 (+ 72us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 11:42:48.403852 (+ 200us) server_negotiation.cc:365] Received JWT_EXCHANGE NegotiatePB request
0620 11:42:48.416580 (+ 12728us) server_negotiation.cc:394] Sending RPC error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:43275/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: certificate has expired
0620 11:42:48.416745 (+ 165us) negotiation.cc:327] Negotiation complete: Not authorized: Server connection negotiation failed: server connection from 127.0.0.1:48458: Failed to load JWKS: Error downloading JWKS from 'https://localhost:43275/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: certificate has expired
Metrics: {"server-negotiator.queue_time_us":26}
W20260620 11:42:48.416900 12262 negotiation.cc:344] Unauthorized connection attempt: Server connection negotiation failed: server connection from 127.0.0.1:48458: Failed to load JWKS: Error downloading JWKS from 'https://localhost:43275/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: certificate has expired
W20260620 11:42:48.417145 12287 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 11:42:48.401007 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.234.254:36633 (local address 127.0.0.1:48458)
0620 11:42:48.401202 (+ 195us) negotiation.cc:107] Waiting for socket to connect
0620 11:42:48.401211 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0620 11:42:48.401270 (+ 59us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 11:42:48.401457 (+ 187us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 11:42:48.401457 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 11:42:48.401459 (+ 2us) client_negotiation.cc:189] Negotiated authn=JWT
0620 11:42:48.401524 (+ 65us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 11:42:48.401526 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 11:42:48.402515 (+ 989us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 11:42:48.402516 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 11:42:48.403141 (+ 625us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 11:42:48.403144 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 11:42:48.403661 (+ 517us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 11:42:48.403662 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 11:42:48.403713 (+ 51us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 11:42:48.403738 (+ 25us) client_negotiation.cc:252] Sending JWT_EXCHANGE NegotiatePB request
0620 11:42:48.416727 (+ 12989us) client_negotiation.cc:283] Received error response from server: Runtime error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:43275/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: certificate has expired
0620 11:42:48.416822 (+ 95us) negotiation.cc:327] Negotiation complete: Runtime error: Client connection negotiation failed: client connection to 127.6.234.254:36633: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:43275/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: certificate has expired
Metrics: {"client-negotiator.queue_time_us":130,"thread_start_us":101,"threads_started":1}
/data0/somelongdirectorytoavoidrpathissues/src/kudu/src/kudu/integration-tests/security-itest.cc:597: Failure
Failed
Bad status: Runtime error: Could not connect to the cluster: Client connection negotiation failed: client connection to 127.6.234.254:36633: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:43275/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: certificate has expired
Google Test trace:
/data0/somelongdirectorytoavoidrpathissues/src/kudu/src/kudu/integration-tests/security-itest.cc:593: Valid JWT
I20260620 11:42:48.417578 7083 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task1JRg7N/build/release/bin/kudu with pid 12191
2026-06-20T11:42:48Z chronyd exiting
I20260620 11:42:48.453598 7083 test_util.cc:175] -----------------------------------------------
I20260620 11:42:48.453783 7083 test_util.cc:176] Had fatal failures, leaving test files at /tmp/dist-test-task1JRg7N/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1781955746769051-7083-0
[ FAILED ] SecurityITest.TestJwtMiniCluster (238 ms)
[ RUN ] SecurityITest.TestJwtMiniClusterWithInvalidCert
I20260620 11:42:48.454586 7083 mini_oidc.cc:152] Starting JWKS server
I20260620 11:42:48.455276 7083 webserver.cc:460] Webserver started at https://127.0.0.1:37019/ using document root <none> and password file <none>
I20260620 11:42:48.455343 7083 mini_oidc.cc:189] Starting OIDC Discovery server
I20260620 11:42:48.455543 7083 webserver.cc:460] Webserver started at http://127.0.0.1:41523/ using document root <none> and password file <none>
2026-06-20T11:42:48Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-20T11:42:48Z Disabled control of system clock
I20260620 11:42:48.463335 7083 external_mini_cluster.cc:1326] Running /tmp/dist-test-task1JRg7N/build/release/bin/kudu
/tmp/dist-test-task1JRg7N/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task1JRg7N/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1781955746769051-7083-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task1JRg7N/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1781955746769051-7083-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task1JRg7N/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1781955746769051-7083-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task1JRg7N/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1781955746769051-7083-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.6.234.254:43349
--webserver_interface=127.6.234.254
--webserver_port=0
--builtin_ntp_servers=127.6.234.212:40891
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.6.234.254:43349
--enable_jwt_token_auth=true
--jwks_url=https://localhost:37019/jwks.json
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--trusted_certificate_file=/tmp/dist-test-task1JRg7N/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1781955746769051-7083-0/testchainca.cert with env {}
W20260620 11:42:48.542680 12294 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 11:42:48.542887 12294 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 11:42:48.542909 12294 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 11:42:48.544073 12294 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 11:42:48.544117 12294 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 11:42:48.544129 12294 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 11:42:48.544181 12294 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 11:42:48.544209 12294 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
W20260620 11:42:48.544236 12294 flags.cc:425] Enabled experimental flag: --enable_jwt_token_auth=true
W20260620 11:42:48.544286 12294 flags.cc:425] Enabled experimental flag: --jwks_url=https://localhost:37019/jwks.json
I20260620 11:42:48.545795 12294 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.6.234.212:40891
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task1JRg7N/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1781955746769051-7083-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task1JRg7N/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1781955746769051-7083-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.6.234.254:43349
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.6.234.254:43349
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--enable_jwt_token_auth=true
--jwks_url=https://localhost:37019/jwks.json
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task1JRg7N/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1781955746769051-7083-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.6.234.254
--webserver_port=0
--trusted_certificate_file=/tmp/dist-test-task1JRg7N/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1781955746769051-7083-0/testchainca.cert
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.12294
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task1JRg7N/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1781955746769051-7083-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 20 Jun 2026 10:32:37 UTC on ccycloud.quasar-bgyofv.root.comops.site
build id 4457173
I20260620 11:42:48.546056 12294 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 11:42:48.546267 12294 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 11:42:48.551041 12302 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 11:42:48.551050 12299 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 11:42:48.551075 12300 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 11:42:48.551249 12294 server_base.cc:1034] running on GCE node
I20260620 11:42:48.551486 12294 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 11:42:48.551767 12294 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 11:42:48.552940 12294 hybrid_clock.cc:648] HybridClock initialized: now 1781955768552926 us; error 28 us; skew 500 ppm
I20260620 11:42:48.554216 12294 webserver.cc:460] Webserver started at http://127.6.234.254:41899/ using document root <none> and password file <none>
I20260620 11:42:48.554458 12294 fs_manager.cc:359] Metadata directory not provided
I20260620 11:42:48.554528 12294 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 11:42:48.554672 12294 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 11:42:48.555696 12294 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task1JRg7N/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1781955746769051-7083-0/minicluster-data/master-0/data/instance:
uuid: "32c2d8cc9a7a4e749ebcf38f4f249e0a"
format_stamp: "Formatted at 2026-06-20 11:42:48 on dist-test-slave-cjc6"
server_key: "4961a354a1c3dc9096faa45b8131b852"
server_key_iv: "d670865484e7eb8a70405221b6c75c44"
server_key_version: "encryptionkey@0"
I20260620 11:42:48.556061 12294 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task1JRg7N/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1781955746769051-7083-0/minicluster-data/master-0/wal/instance:
uuid: "32c2d8cc9a7a4e749ebcf38f4f249e0a"
format_stamp: "Formatted at 2026-06-20 11:42:48 on dist-test-slave-cjc6"
server_key: "4961a354a1c3dc9096faa45b8131b852"
server_key_iv: "d670865484e7eb8a70405221b6c75c44"
server_key_version: "encryptionkey@0"
I20260620 11:42:48.557390 12294 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.001s
I20260620 11:42:48.558138 12308 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 11:42:48.558308 12294 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.000s
I20260620 11:42:48.558415 12294 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task1JRg7N/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1781955746769051-7083-0/minicluster-data/master-0/data,/tmp/dist-test-task1JRg7N/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1781955746769051-7083-0/minicluster-data/master-0/wal
uuid: "32c2d8cc9a7a4e749ebcf38f4f249e0a"
format_stamp: "Formatted at 2026-06-20 11:42:48 on dist-test-slave-cjc6"
server_key: "4961a354a1c3dc9096faa45b8131b852"
server_key_iv: "d670865484e7eb8a70405221b6c75c44"
server_key_version: "encryptionkey@0"
I20260620 11:42:48.558488 12294 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task1JRg7N/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1781955746769051-7083-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task1JRg7N/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1781955746769051-7083-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task1JRg7N/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1781955746769051-7083-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 11:42:48.569088 12294 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 11:42:48.569376 12294 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 11:42:48.569526 12294 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 11:42:48.573113 12294 rpc_server.cc:307] RPC server started. Bound to: 127.6.234.254:43349
I20260620 11:42:48.573144 12360 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.6.234.254:43349 every 8 connection(s)
I20260620 11:42:48.573473 12294 server_base.cc:1166] Dumped server information to /tmp/dist-test-task1JRg7N/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1781955746769051-7083-0/minicluster-data/master-0/data/info.pb
I20260620 11:42:48.574396 12361 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 11:42:48.576511 12361 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 32c2d8cc9a7a4e749ebcf38f4f249e0a: Bootstrap starting.
I20260620 11:42:48.577015 12361 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 32c2d8cc9a7a4e749ebcf38f4f249e0a: Neither blocks nor log segments found. Creating new log.
I20260620 11:42:48.577237 12361 log.cc:824] T 00000000000000000000000000000000 P 32c2d8cc9a7a4e749ebcf38f4f249e0a: Log is configured to *not* fsync() on all Append() calls
I20260620 11:42:48.577414 7083 external_mini_cluster.cc:1388] Started /tmp/dist-test-task1JRg7N/build/release/bin/kudu as pid 12294
I20260620 11:42:48.577534 7083 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task1JRg7N/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1781955746769051-7083-0/minicluster-data/master-0/wal/instance
I20260620 11:42:48.577744 7083 external_mini_cluster.cc:1428] Setting key 634b897e8be9f6babcd08e71ab1b9278
I20260620 11:42:48.578032 12361 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 32c2d8cc9a7a4e749ebcf38f4f249e0a: No bootstrap required, opened a new log
I20260620 11:42:48.579840 12361 raft_consensus.cc:348] T 00000000000000000000000000000000 P 32c2d8cc9a7a4e749ebcf38f4f249e0a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "32c2d8cc9a7a4e749ebcf38f4f249e0a" member_type: VOTER last_known_addr { host: "127.6.234.254" port: 43349 } }
I20260620 11:42:48.579984 12361 raft_consensus.cc:374] T 00000000000000000000000000000000 P 32c2d8cc9a7a4e749ebcf38f4f249e0a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 11:42:48.580008 12361 raft_consensus.cc:729] T 00000000000000000000000000000000 P 32c2d8cc9a7a4e749ebcf38f4f249e0a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 32c2d8cc9a7a4e749ebcf38f4f249e0a, State: Initialized, Role: FOLLOWER
I20260620 11:42:48.580138 12361 consensus_queue.cc:260] T 00000000000000000000000000000000 P 32c2d8cc9a7a4e749ebcf38f4f249e0a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "32c2d8cc9a7a4e749ebcf38f4f249e0a" member_type: VOTER last_known_addr { host: "127.6.234.254" port: 43349 } }
I20260620 11:42:48.580210 12361 raft_consensus.cc:388] T 00000000000000000000000000000000 P 32c2d8cc9a7a4e749ebcf38f4f249e0a [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260620 11:42:48.580231 12361 raft_consensus.cc:482] T 00000000000000000000000000000000 P 32c2d8cc9a7a4e749ebcf38f4f249e0a [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260620 11:42:48.580304 12361 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 32c2d8cc9a7a4e749ebcf38f4f249e0a [term 0 FOLLOWER]: Advancing to term 1
I20260620 11:42:48.580832 12361 raft_consensus.cc:504] T 00000000000000000000000000000000 P 32c2d8cc9a7a4e749ebcf38f4f249e0a [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "32c2d8cc9a7a4e749ebcf38f4f249e0a" member_type: VOTER last_known_addr { host: "127.6.234.254" port: 43349 } }
I20260620 11:42:48.580924 12361 leader_election.cc:302] T 00000000000000000000000000000000 P 32c2d8cc9a7a4e749ebcf38f4f249e0a [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 32c2d8cc9a7a4e749ebcf38f4f249e0a; no voters:
I20260620 11:42:48.581097 12361 leader_election.cc:288] T 00000000000000000000000000000000 P 32c2d8cc9a7a4e749ebcf38f4f249e0a [CANDIDATE]: Term 1 election: Requested vote from peers
I20260620 11:42:48.581118 12366 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 32c2d8cc9a7a4e749ebcf38f4f249e0a [term 1 FOLLOWER]: Leader election won for term 1
I20260620 11:42:48.581225 12366 raft_consensus.cc:686] T 00000000000000000000000000000000 P 32c2d8cc9a7a4e749ebcf38f4f249e0a [term 1 LEADER]: Becoming Leader. State: Replica: 32c2d8cc9a7a4e749ebcf38f4f249e0a, State: Running, Role: LEADER
I20260620 11:42:48.581308 12366 consensus_queue.cc:237] T 00000000000000000000000000000000 P 32c2d8cc9a7a4e749ebcf38f4f249e0a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "32c2d8cc9a7a4e749ebcf38f4f249e0a" member_type: VOTER last_known_addr { host: "127.6.234.254" port: 43349 } }
I20260620 11:42:48.581347 12361 sys_catalog.cc:564] T 00000000000000000000000000000000 P 32c2d8cc9a7a4e749ebcf38f4f249e0a [sys.catalog]: configured and running, proceeding with master startup.
I20260620 11:42:48.581610 12367 sys_catalog.cc:455] T 00000000000000000000000000000000 P 32c2d8cc9a7a4e749ebcf38f4f249e0a [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "32c2d8cc9a7a4e749ebcf38f4f249e0a" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "32c2d8cc9a7a4e749ebcf38f4f249e0a" member_type: VOTER last_known_addr { host: "127.6.234.254" port: 43349 } } }
I20260620 11:42:48.581722 12367 sys_catalog.cc:458] T 00000000000000000000000000000000 P 32c2d8cc9a7a4e749ebcf38f4f249e0a [sys.catalog]: This master's current role is: LEADER
I20260620 11:42:48.581897 12368 sys_catalog.cc:455] T 00000000000000000000000000000000 P 32c2d8cc9a7a4e749ebcf38f4f249e0a [sys.catalog]: SysCatalogTable state changed. Reason: New leader 32c2d8cc9a7a4e749ebcf38f4f249e0a. Latest consensus state: current_term: 1 leader_uuid: "32c2d8cc9a7a4e749ebcf38f4f249e0a" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "32c2d8cc9a7a4e749ebcf38f4f249e0a" member_type: VOTER last_known_addr { host: "127.6.234.254" port: 43349 } } }
I20260620 11:42:48.581959 12368 sys_catalog.cc:458] T 00000000000000000000000000000000 P 32c2d8cc9a7a4e749ebcf38f4f249e0a [sys.catalog]: This master's current role is: LEADER
I20260620 11:42:48.582032 12371 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260620 11:42:48.582576 12371 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260620 11:42:48.584436 12365 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 11:42:48.578211 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37682 (local address 127.6.234.254:43349)
0620 11:42:48.578440 (+ 229us) server_negotiation.cc:206] Beginning negotiation
0620 11:42:48.578447 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0620 11:42:48.578470 (+ 23us) server_negotiation.cc:407] Connection header received
0620 11:42:48.579196 (+ 726us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 11:42:48.579203 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 11:42:48.579268 (+ 65us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 11:42:48.579418 (+ 150us) server_negotiation.cc:226] Negotiated authn=SASL
0620 11:42:48.579624 (+ 206us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 11:42:48.580925 (+ 1301us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 11:42:48.582047 (+ 1122us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 11:42:48.582618 (+ 571us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 11:42:48.582715 (+ 97us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 11:42:48.583697 (+ 982us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 11:42:48.583701 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 11:42:48.583701 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 11:42:48.583909 (+ 208us) server_negotiation.cc:922] calling sasl_server_start()
0620 11:42:48.583952 (+ 43us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 11:42:48.583966 (+ 14us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 11:42:48.584006 (+ 40us) server_negotiation.cc:1035] Waiting for connection context
0620 11:42:48.584143 (+ 137us) server_negotiation.cc:299] Negotiation successful
0620 11:42:48.584235 (+ 92us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":142,"thread_start_us":85,"threads_started":1}
I20260620 11:42:48.585160 12371 catalog_manager.cc:1347] Generated new cluster ID: 823bccd3c4f9461985621b7bbba56a2d
I20260620 11:42:48.585215 12371 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260620 11:42:48.585343 7083 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260620 11:42:48.607136 12371 catalog_manager.cc:1370] Generated new certificate authority record
I20260620 11:42:48.607857 12371 catalog_manager.cc:1504] Loading token signing keys...
I20260620 11:42:48.611464 12371 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 32c2d8cc9a7a4e749ebcf38f4f249e0a: Generated new TSK 0
I20260620 11:42:48.611644 12371 catalog_manager.cc:1514] Initializing in-progress tserver states...
E20260620 11:42:48.635185 12391 webserver.cc:533] Webserver: sslize failed: error:14094418:SSL routines:ssl3_read_bytes:tlsv1 alert unknown ca
E20260620 11:42:48.635265 12391 webserver.cc:533] Webserver: error shutting down SSL connection: error:140E0197:SSL routines:SSL_shutdown:shutdown while in init
I20260620 11:42:48.635650 12365 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 11:42:48.621546 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37696 (local address 127.6.234.254:43349)
0620 11:42:48.621616 (+ 70us) server_negotiation.cc:206] Beginning negotiation
0620 11:42:48.621618 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 11:42:48.621756 (+ 138us) server_negotiation.cc:407] Connection header received
0620 11:42:48.621845 (+ 89us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 11:42:48.621846 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 11:42:48.621858 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 11:42:48.621928 (+ 70us) server_negotiation.cc:226] Negotiated authn=JWT
0620 11:42:48.622072 (+ 144us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 11:42:48.622606 (+ 534us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 11:42:48.623382 (+ 776us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 11:42:48.623604 (+ 222us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 11:42:48.623683 (+ 79us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 11:42:48.623883 (+ 200us) server_negotiation.cc:365] Received JWT_EXCHANGE NegotiatePB request
0620 11:42:48.635407 (+ 11524us) server_negotiation.cc:394] Sending RPC error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:37019/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: unable to get local issuer certificate
0620 11:42:48.635565 (+ 158us) negotiation.cc:327] Negotiation complete: Not authorized: Server connection negotiation failed: server connection from 127.0.0.1:37696: Failed to load JWKS: Error downloading JWKS from 'https://localhost:37019/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: unable to get local issuer certificate
Metrics: {"server-negotiator.queue_time_us":44}
W20260620 11:42:48.635711 12365 negotiation.cc:344] Unauthorized connection attempt: Server connection negotiation failed: server connection from 127.0.0.1:37696: Failed to load JWKS: Error downloading JWKS from 'https://localhost:37019/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: unable to get local issuer certificate
W20260620 11:42:48.635728 12390 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 11:42:48.621475 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.234.254:43349 (local address 127.0.0.1:37696)
0620 11:42:48.621659 (+ 184us) negotiation.cc:107] Waiting for socket to connect
0620 11:42:48.621668 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0620 11:42:48.621774 (+ 106us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 11:42:48.621934 (+ 160us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 11:42:48.621935 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 11:42:48.621936 (+ 1us) client_negotiation.cc:189] Negotiated authn=JWT
0620 11:42:48.621995 (+ 59us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 11:42:48.621997 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 11:42:48.622689 (+ 692us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 11:42:48.622690 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 11:42:48.623294 (+ 604us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 11:42:48.623296 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 11:42:48.623683 (+ 387us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 11:42:48.623684 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 11:42:48.623751 (+ 67us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 11:42:48.623756 (+ 5us) client_negotiation.cc:252] Sending JWT_EXCHANGE NegotiatePB request
0620 11:42:48.635512 (+ 11756us) client_negotiation.cc:283] Received error response from server: Runtime error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:37019/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: unable to get local issuer certificate
0620 11:42:48.635615 (+ 103us) negotiation.cc:327] Negotiation complete: Runtime error: Client connection negotiation failed: client connection to 127.6.234.254:43349: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:37019/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: unable to get local issuer certificate
Metrics: {"client-negotiator.queue_time_us":163,"thread_start_us":108,"threads_started":1}
I20260620 11:42:48.636092 7083 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task1JRg7N/build/release/bin/kudu with pid 12294
2026-06-20T11:42:48Z chronyd exiting
[ OK ] SecurityITest.TestJwtMiniClusterWithInvalidCert (219 ms)
[ RUN ] SecurityITest.TestJwtMiniClusterWithUntrustedCert
I20260620 11:42:48.673882 7083 mini_oidc.cc:152] Starting JWKS server
I20260620 11:42:48.674574 7083 webserver.cc:460] Webserver started at https://127.0.0.1:34241/ using document root <none> and password file <none>
I20260620 11:42:48.674615 7083 mini_oidc.cc:189] Starting OIDC Discovery server
I20260620 11:42:48.674779 7083 webserver.cc:460] Webserver started at http://127.0.0.1:33691/ using document root <none> and password file <none>
2026-06-20T11:42:48Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-20T11:42:48Z Disabled control of system clock
I20260620 11:42:48.682497 7083 external_mini_cluster.cc:1326] Running /tmp/dist-test-task1JRg7N/build/release/bin/kudu
/tmp/dist-test-task1JRg7N/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task1JRg7N/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1781955746769051-7083-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task1JRg7N/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1781955746769051-7083-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task1JRg7N/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1781955746769051-7083-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task1JRg7N/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1781955746769051-7083-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.6.234.254:38455
--webserver_interface=127.6.234.254
--webserver_port=0
--builtin_ntp_servers=127.6.234.212:46155
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.6.234.254:38455
--enable_jwt_token_auth=true
--jwks_url=https://localhost:34241/jwks.json
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {}
W20260620 11:42:48.764606 12397 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 11:42:48.764797 12397 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 11:42:48.764817 12397 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 11:42:48.766072 12397 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 11:42:48.766113 12397 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 11:42:48.766126 12397 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 11:42:48.766139 12397 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 11:42:48.766151 12397 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
W20260620 11:42:48.766163 12397 flags.cc:425] Enabled experimental flag: --enable_jwt_token_auth=true
W20260620 11:42:48.766181 12397 flags.cc:425] Enabled experimental flag: --jwks_url=https://localhost:34241/jwks.json
I20260620 11:42:48.767482 12397 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.6.234.212:46155
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task1JRg7N/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1781955746769051-7083-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task1JRg7N/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1781955746769051-7083-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.6.234.254:38455
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.6.234.254:38455
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--enable_jwt_token_auth=true
--jwks_url=https://localhost:34241/jwks.json
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task1JRg7N/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1781955746769051-7083-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.6.234.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.12397
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task1JRg7N/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1781955746769051-7083-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 20 Jun 2026 10:32:37 UTC on ccycloud.quasar-bgyofv.root.comops.site
build id 4457173
I20260620 11:42:48.767670 12397 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 11:42:48.767890 12397 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 11:42:48.772590 12405 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 11:42:48.772568 12402 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 11:42:48.772617 12403 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 11:42:48.772790 12397 server_base.cc:1034] running on GCE node
I20260620 11:42:48.773167 12397 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 11:42:48.773468 12397 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 11:42:48.774632 12397 hybrid_clock.cc:648] HybridClock initialized: now 1781955768774607 us; error 41 us; skew 500 ppm
I20260620 11:42:48.775836 12397 webserver.cc:460] Webserver started at http://127.6.234.254:38849/ using document root <none> and password file <none>
I20260620 11:42:48.776126 12397 fs_manager.cc:359] Metadata directory not provided
I20260620 11:42:48.776194 12397 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 11:42:48.776340 12397 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 11:42:48.777377 12397 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task1JRg7N/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1781955746769051-7083-0/minicluster-data/master-0/data/instance:
uuid: "9235543c314d4a7c878212cdf92adec1"
format_stamp: "Formatted at 2026-06-20 11:42:48 on dist-test-slave-cjc6"
server_key: "362ee224b0d2e8fcafe8c6567021e8a5"
server_key_iv: "02df062fded7367068109a592d283ce2"
server_key_version: "encryptionkey@0"
I20260620 11:42:48.777801 12397 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task1JRg7N/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1781955746769051-7083-0/minicluster-data/master-0/wal/instance:
uuid: "9235543c314d4a7c878212cdf92adec1"
format_stamp: "Formatted at 2026-06-20 11:42:48 on dist-test-slave-cjc6"
server_key: "362ee224b0d2e8fcafe8c6567021e8a5"
server_key_iv: "02df062fded7367068109a592d283ce2"
server_key_version: "encryptionkey@0"
I20260620 11:42:48.779091 12397 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.001s
I20260620 11:42:48.779827 12411 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 11:42:48.780006 12397 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.001s
I20260620 11:42:48.780103 12397 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task1JRg7N/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1781955746769051-7083-0/minicluster-data/master-0/data,/tmp/dist-test-task1JRg7N/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1781955746769051-7083-0/minicluster-data/master-0/wal
uuid: "9235543c314d4a7c878212cdf92adec1"
format_stamp: "Formatted at 2026-06-20 11:42:48 on dist-test-slave-cjc6"
server_key: "362ee224b0d2e8fcafe8c6567021e8a5"
server_key_iv: "02df062fded7367068109a592d283ce2"
server_key_version: "encryptionkey@0"
I20260620 11:42:48.780190 12397 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task1JRg7N/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1781955746769051-7083-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task1JRg7N/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1781955746769051-7083-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task1JRg7N/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1781955746769051-7083-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 11:42:48.805773 12397 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 11:42:48.806130 12397 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 11:42:48.806291 12397 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 11:42:48.810321 12397 rpc_server.cc:307] RPC server started. Bound to: 127.6.234.254:38455
I20260620 11:42:48.810379 12463 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.6.234.254:38455 every 8 connection(s)
I20260620 11:42:48.810761 12397 server_base.cc:1166] Dumped server information to /tmp/dist-test-task1JRg7N/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1781955746769051-7083-0/minicluster-data/master-0/data/info.pb
I20260620 11:42:48.811419 12464 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 11:42:48.814167 12464 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 9235543c314d4a7c878212cdf92adec1: Bootstrap starting.
I20260620 11:42:48.814754 12464 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 9235543c314d4a7c878212cdf92adec1: Neither blocks nor log segments found. Creating new log.
I20260620 11:42:48.815038 12464 log.cc:824] T 00000000000000000000000000000000 P 9235543c314d4a7c878212cdf92adec1: Log is configured to *not* fsync() on all Append() calls
I20260620 11:42:48.815804 12464 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 9235543c314d4a7c878212cdf92adec1: No bootstrap required, opened a new log
I20260620 11:42:48.816749 7083 external_mini_cluster.cc:1388] Started /tmp/dist-test-task1JRg7N/build/release/bin/kudu as pid 12397
I20260620 11:42:48.816862 7083 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task1JRg7N/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1781955746769051-7083-0/minicluster-data/master-0/wal/instance
I20260620 11:42:48.817075 7083 external_mini_cluster.cc:1428] Setting key 1c04c80e9af8c2d685c2ec7c5a0bc28f
I20260620 11:42:48.818173 12464 raft_consensus.cc:348] T 00000000000000000000000000000000 P 9235543c314d4a7c878212cdf92adec1 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9235543c314d4a7c878212cdf92adec1" member_type: VOTER last_known_addr { host: "127.6.234.254" port: 38455 } }
I20260620 11:42:48.818357 12464 raft_consensus.cc:374] T 00000000000000000000000000000000 P 9235543c314d4a7c878212cdf92adec1 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 11:42:48.818395 12464 raft_consensus.cc:729] T 00000000000000000000000000000000 P 9235543c314d4a7c878212cdf92adec1 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9235543c314d4a7c878212cdf92adec1, State: Initialized, Role: FOLLOWER
I20260620 11:42:48.818531 12464 consensus_queue.cc:260] T 00000000000000000000000000000000 P 9235543c314d4a7c878212cdf92adec1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9235543c314d4a7c878212cdf92adec1" member_type: VOTER last_known_addr { host: "127.6.234.254" port: 38455 } }
I20260620 11:42:48.818639 12464 raft_consensus.cc:388] T 00000000000000000000000000000000 P 9235543c314d4a7c878212cdf92adec1 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260620 11:42:48.818671 12464 raft_consensus.cc:482] T 00000000000000000000000000000000 P 9235543c314d4a7c878212cdf92adec1 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260620 11:42:48.818727 12464 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 9235543c314d4a7c878212cdf92adec1 [term 0 FOLLOWER]: Advancing to term 1
I20260620 11:42:48.819485 12464 raft_consensus.cc:504] T 00000000000000000000000000000000 P 9235543c314d4a7c878212cdf92adec1 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9235543c314d4a7c878212cdf92adec1" member_type: VOTER last_known_addr { host: "127.6.234.254" port: 38455 } }
I20260620 11:42:48.819597 12464 leader_election.cc:302] T 00000000000000000000000000000000 P 9235543c314d4a7c878212cdf92adec1 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 9235543c314d4a7c878212cdf92adec1; no voters:
I20260620 11:42:48.819741 12464 leader_election.cc:288] T 00000000000000000000000000000000 P 9235543c314d4a7c878212cdf92adec1 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260620 11:42:48.819881 12469 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 9235543c314d4a7c878212cdf92adec1 [term 1 FOLLOWER]: Leader election won for term 1
I20260620 11:42:48.820029 12469 raft_consensus.cc:686] T 00000000000000000000000000000000 P 9235543c314d4a7c878212cdf92adec1 [term 1 LEADER]: Becoming Leader. State: Replica: 9235543c314d4a7c878212cdf92adec1, State: Running, Role: LEADER
I20260620 11:42:48.820013 12464 sys_catalog.cc:564] T 00000000000000000000000000000000 P 9235543c314d4a7c878212cdf92adec1 [sys.catalog]: configured and running, proceeding with master startup.
I20260620 11:42:48.820168 12469 consensus_queue.cc:237] T 00000000000000000000000000000000 P 9235543c314d4a7c878212cdf92adec1 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9235543c314d4a7c878212cdf92adec1" member_type: VOTER last_known_addr { host: "127.6.234.254" port: 38455 } }
I20260620 11:42:48.820771 12471 sys_catalog.cc:455] T 00000000000000000000000000000000 P 9235543c314d4a7c878212cdf92adec1 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 9235543c314d4a7c878212cdf92adec1. Latest consensus state: current_term: 1 leader_uuid: "9235543c314d4a7c878212cdf92adec1" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9235543c314d4a7c878212cdf92adec1" member_type: VOTER last_known_addr { host: "127.6.234.254" port: 38455 } } }
I20260620 11:42:48.820883 12471 sys_catalog.cc:458] T 00000000000000000000000000000000 P 9235543c314d4a7c878212cdf92adec1 [sys.catalog]: This master's current role is: LEADER
I20260620 11:42:48.820757 12470 sys_catalog.cc:455] T 00000000000000000000000000000000 P 9235543c314d4a7c878212cdf92adec1 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "9235543c314d4a7c878212cdf92adec1" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9235543c314d4a7c878212cdf92adec1" member_type: VOTER last_known_addr { host: "127.6.234.254" port: 38455 } } }
I20260620 11:42:48.821206 12470 sys_catalog.cc:458] T 00000000000000000000000000000000 P 9235543c314d4a7c878212cdf92adec1 [sys.catalog]: This master's current role is: LEADER
I20260620 11:42:48.821398 12478 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260620 11:42:48.822165 12478 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260620 11:42:48.823877 12468 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 11:42:48.817571 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38888 (local address 127.6.234.254:38455)
0620 11:42:48.817844 (+ 273us) server_negotiation.cc:206] Beginning negotiation
0620 11:42:48.817855 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0620 11:42:48.817895 (+ 40us) server_negotiation.cc:407] Connection header received
0620 11:42:48.818127 (+ 232us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 11:42:48.818135 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 11:42:48.818224 (+ 89us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 11:42:48.818375 (+ 151us) server_negotiation.cc:226] Negotiated authn=SASL
0620 11:42:48.818644 (+ 269us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 11:42:48.820471 (+ 1827us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 11:42:48.821650 (+ 1179us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 11:42:48.822231 (+ 581us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 11:42:48.822353 (+ 122us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 11:42:48.823173 (+ 820us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 11:42:48.823177 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 11:42:48.823177 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 11:42:48.823340 (+ 163us) server_negotiation.cc:922] calling sasl_server_start()
0620 11:42:48.823373 (+ 33us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 11:42:48.823384 (+ 11us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 11:42:48.823439 (+ 55us) server_negotiation.cc:1035] Waiting for connection context
0620 11:42:48.823533 (+ 94us) server_negotiation.cc:299] Negotiation successful
0620 11:42:48.823621 (+ 88us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":183,"thread_start_us":100,"threads_started":1}
I20260620 11:42:48.824909 7083 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260620 11:42:48.826718 12478 catalog_manager.cc:1347] Generated new cluster ID: 0d46c7a1e1594247a0167937a1316092
I20260620 11:42:48.826774 12478 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260620 11:42:48.844580 12478 catalog_manager.cc:1370] Generated new certificate authority record
I20260620 11:42:48.845341 12478 catalog_manager.cc:1504] Loading token signing keys...
I20260620 11:42:48.850395 12478 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 9235543c314d4a7c878212cdf92adec1: Generated new TSK 0
I20260620 11:42:48.850596 12478 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260620 11:42:48.867697 12468 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 11:42:48.863794 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38904 (local address 127.6.234.254:38455)
0620 11:42:48.863861 (+ 67us) server_negotiation.cc:206] Beginning negotiation
0620 11:42:48.863864 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0620 11:42:48.864004 (+ 140us) server_negotiation.cc:407] Connection header received
0620 11:42:48.864115 (+ 111us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 11:42:48.864116 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 11:42:48.864127 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 11:42:48.864182 (+ 55us) server_negotiation.cc:226] Negotiated authn=JWT
0620 11:42:48.864326 (+ 144us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 11:42:48.864796 (+ 470us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 11:42:48.866154 (+ 1358us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 11:42:48.866338 (+ 184us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 11:42:48.866390 (+ 52us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 11:42:48.866574 (+ 184us) server_negotiation.cc:365] Received JWT_EXCHANGE NegotiatePB request
0620 11:42:48.867488 (+ 914us) server_negotiation.cc:394] Sending RPC error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:34241/jwks.json': curl error: Problem with the SSL CA cert (path? access rights?): error setting certificate verify locations:
CAfile: /etc/pki/tls/certs/ca-bundle.crt
CApath: none
0620 11:42:48.867616 (+ 128us) negotiation.cc:327] Negotiation complete: Not authorized: Server connection negotiation failed: server connection from 127.0.0.1:38904: Failed to load JWKS: Error downloading JWKS from 'https://localhost:34241/jwks.json': curl error: Problem with the SSL CA cert (path? access rights?): error setting certificate verify locations:
CAfile: /etc/pki/tls/certs/ca-bundle.crt
CApath: none
Metrics: {"server-negotiator.queue_time_us":38}
W20260620 11:42:48.867760 12468 negotiation.cc:344] Unauthorized connection attempt: Server connection negotiation failed: server connection from 127.0.0.1:38904: Failed to load JWKS: Error downloading JWKS from 'https://localhost:34241/jwks.json': curl error: Problem with the SSL CA cert (path? access rights?): error setting certificate verify locations:
CAfile: /etc/pki/tls/certs/ca-bundle.crt
CApath: none
W20260620 11:42:48.867794 12493 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 11:42:48.863743 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.234.254:38455 (local address 127.0.0.1:38904)
0620 11:42:48.863932 (+ 189us) negotiation.cc:107] Waiting for socket to connect
0620 11:42:48.863943 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0620 11:42:48.864009 (+ 66us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 11:42:48.864198 (+ 189us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 11:42:48.864198 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 11:42:48.864200 (+ 2us) client_negotiation.cc:189] Negotiated authn=JWT
0620 11:42:48.864255 (+ 55us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 11:42:48.864257 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 11:42:48.865095 (+ 838us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 11:42:48.865096 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 11:42:48.866077 (+ 981us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 11:42:48.866080 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 11:42:48.866413 (+ 333us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 11:42:48.866414 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 11:42:48.866479 (+ 65us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 11:42:48.866484 (+ 5us) client_negotiation.cc:252] Sending JWT_EXCHANGE NegotiatePB request
0620 11:42:48.867594 (+ 1110us) client_negotiation.cc:283] Received error response from server: Runtime error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:34241/jwks.json': curl error: Problem with the SSL CA cert (path? access rights?): error setting certificate verify locations:
CAfile: /etc/pki/tls/certs/ca-bundle.crt
CApath: none
0620 11:42:48.867691 (+ 97us) negotiation.cc:327] Negotiation complete: Runtime error: Client connection negotiation failed: client connection to 127.6.234.254:38455: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:34241/jwks.json': curl error: Problem with the SSL CA cert (path? access rights?): error setting certificate verify locations:
CAfile: /etc/pki/tls/certs/ca-bundle.crt
CApath: none
Metrics: {"client-negotiator.queue_time_us":164,"thread_start_us":121,"threads_started":1}
I20260620 11:42:48.868009 7083 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task1JRg7N/build/release/bin/kudu with pid 12397
E20260620 11:42:48.878333 12494 webserver.cc:533] Webserver: sslize failed:
E20260620 11:42:48.878435 12494 webserver.cc:533] Webserver: error shutting down SSL connection: error:140E0197:SSL routines:SSL_shutdown:shutdown while in init
2026-06-20T11:42:48Z chronyd exiting
[ OK ] SecurityITest.TestJwtMiniClusterWithUntrustedCert (231 ms)
[ RUN ] SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS
2026-06-20T11:42:48Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-20T11:42:48Z Disabled control of system clock
I20260620 11:42:48.912828 7083 external_mini_cluster.cc:1326] Running /tmp/dist-test-task1JRg7N/build/release/bin/kudu
/tmp/dist-test-task1JRg7N/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task1JRg7N/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1781955746769051-7083-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task1JRg7N/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1781955746769051-7083-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task1JRg7N/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1781955746769051-7083-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task1JRg7N/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1781955746769051-7083-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.6.234.254:36349
--webserver_interface=127.6.234.254
--webserver_port=0
--builtin_ntp_servers=127.6.234.212:34689
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.6.234.254:36349
--enable_jwt_token_auth=true
--jwks_url=default.url
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--trusted_certificate_file=/tmp/dist-test-task1JRg7N/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1781955746769051-7083-0/testchainca.cert with env {}
W20260620 11:42:48.992254 12498 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 11:42:48.992462 12498 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 11:42:48.992484 12498 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 11:42:48.993562 12498 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 11:42:48.993594 12498 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 11:42:48.993607 12498 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 11:42:48.993620 12498 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 11:42:48.993633 12498 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
W20260620 11:42:48.993644 12498 flags.cc:425] Enabled experimental flag: --enable_jwt_token_auth=true
W20260620 11:42:48.993655 12498 flags.cc:425] Enabled experimental flag: --jwks_url=default.url
I20260620 11:42:48.995056 12498 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.6.234.212:34689
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task1JRg7N/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1781955746769051-7083-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task1JRg7N/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1781955746769051-7083-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.6.234.254:36349
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.6.234.254:36349
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--enable_jwt_token_auth=true
--jwks_url=default.url
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task1JRg7N/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1781955746769051-7083-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.6.234.254
--webserver_port=0
--trusted_certificate_file=/tmp/dist-test-task1JRg7N/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1781955746769051-7083-0/testchainca.cert
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.12498
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task1JRg7N/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1781955746769051-7083-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 20 Jun 2026 10:32:37 UTC on ccycloud.quasar-bgyofv.root.comops.site
build id 4457173
I20260620 11:42:48.995245 12498 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 11:42:48.995400 12498 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260620 11:42:49.000154 12498 server_base.cc:1034] running on GCE node
W20260620 11:42:49.000304 12503 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 11:42:49.000334 12504 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 11:42:49.000442 12506 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 11:42:49.000661 12498 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 11:42:49.000926 12498 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 11:42:49.002095 12498 hybrid_clock.cc:648] HybridClock initialized: now 1781955769002079 us; error 45 us; skew 500 ppm
I20260620 11:42:49.003330 12498 webserver.cc:460] Webserver started at http://127.6.234.254:43273/ using document root <none> and password file <none>
I20260620 11:42:49.003566 12498 fs_manager.cc:359] Metadata directory not provided
I20260620 11:42:49.003634 12498 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 11:42:49.003826 12498 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 11:42:49.004878 12498 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task1JRg7N/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1781955746769051-7083-0/minicluster-data/master-0/data/instance:
uuid: "4c0cd57525104ecdaed9a1dfc24d67ae"
format_stamp: "Formatted at 2026-06-20 11:42:48 on dist-test-slave-cjc6"
server_key: "1550c7876e1804d6c5ae785dd02bb322"
server_key_iv: "de56a043417251e5328cc5eeda68eee8"
server_key_version: "encryptionkey@0"
I20260620 11:42:49.005235 12498 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task1JRg7N/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1781955746769051-7083-0/minicluster-data/master-0/wal/instance:
uuid: "4c0cd57525104ecdaed9a1dfc24d67ae"
format_stamp: "Formatted at 2026-06-20 11:42:48 on dist-test-slave-cjc6"
server_key: "1550c7876e1804d6c5ae785dd02bb322"
server_key_iv: "de56a043417251e5328cc5eeda68eee8"
server_key_version: "encryptionkey@0"
I20260620 11:42:49.006644 12498 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.003s
I20260620 11:42:49.007480 12512 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 11:42:49.007694 12498 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.001s
I20260620 11:42:49.007800 12498 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task1JRg7N/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1781955746769051-7083-0/minicluster-data/master-0/data,/tmp/dist-test-task1JRg7N/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1781955746769051-7083-0/minicluster-data/master-0/wal
uuid: "4c0cd57525104ecdaed9a1dfc24d67ae"
format_stamp: "Formatted at 2026-06-20 11:42:48 on dist-test-slave-cjc6"
server_key: "1550c7876e1804d6c5ae785dd02bb322"
server_key_iv: "de56a043417251e5328cc5eeda68eee8"
server_key_version: "encryptionkey@0"
I20260620 11:42:49.007898 12498 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task1JRg7N/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1781955746769051-7083-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task1JRg7N/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1781955746769051-7083-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task1JRg7N/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1781955746769051-7083-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 11:42:49.029784 12498 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 11:42:49.030120 12498 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 11:42:49.030285 12498 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 11:42:49.034169 12498 rpc_server.cc:307] RPC server started. Bound to: 127.6.234.254:36349
I20260620 11:42:49.034210 12564 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.6.234.254:36349 every 8 connection(s)
I20260620 11:42:49.034574 12498 server_base.cc:1166] Dumped server information to /tmp/dist-test-task1JRg7N/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1781955746769051-7083-0/minicluster-data/master-0/data/info.pb
I20260620 11:42:49.035522 12565 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 11:42:49.037071 7083 external_mini_cluster.cc:1388] Started /tmp/dist-test-task1JRg7N/build/release/bin/kudu as pid 12498
I20260620 11:42:49.037186 7083 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task1JRg7N/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1781955746769051-7083-0/minicluster-data/master-0/wal/instance
I20260620 11:42:49.037351 7083 external_mini_cluster.cc:1428] Setting key 3f7aedad44322efcef845277fa019908
I20260620 11:42:49.038241 12565 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 4c0cd57525104ecdaed9a1dfc24d67ae: Bootstrap starting.
I20260620 11:42:49.038978 12565 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 4c0cd57525104ecdaed9a1dfc24d67ae: Neither blocks nor log segments found. Creating new log.
I20260620 11:42:49.039319 12565 log.cc:824] T 00000000000000000000000000000000 P 4c0cd57525104ecdaed9a1dfc24d67ae: Log is configured to *not* fsync() on all Append() calls
I20260620 11:42:49.040051 12565 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 4c0cd57525104ecdaed9a1dfc24d67ae: No bootstrap required, opened a new log
I20260620 11:42:49.042201 12565 raft_consensus.cc:348] T 00000000000000000000000000000000 P 4c0cd57525104ecdaed9a1dfc24d67ae [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4c0cd57525104ecdaed9a1dfc24d67ae" member_type: VOTER last_known_addr { host: "127.6.234.254" port: 36349 } }
I20260620 11:42:49.042384 12565 raft_consensus.cc:374] T 00000000000000000000000000000000 P 4c0cd57525104ecdaed9a1dfc24d67ae [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 11:42:49.042428 12565 raft_consensus.cc:729] T 00000000000000000000000000000000 P 4c0cd57525104ecdaed9a1dfc24d67ae [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4c0cd57525104ecdaed9a1dfc24d67ae, State: Initialized, Role: FOLLOWER
I20260620 11:42:49.042554 12565 consensus_queue.cc:260] T 00000000000000000000000000000000 P 4c0cd57525104ecdaed9a1dfc24d67ae [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4c0cd57525104ecdaed9a1dfc24d67ae" member_type: VOTER last_known_addr { host: "127.6.234.254" port: 36349 } }
I20260620 11:42:49.042644 12565 raft_consensus.cc:388] T 00000000000000000000000000000000 P 4c0cd57525104ecdaed9a1dfc24d67ae [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260620 11:42:49.042675 12565 raft_consensus.cc:482] T 00000000000000000000000000000000 P 4c0cd57525104ecdaed9a1dfc24d67ae [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260620 11:42:49.042718 12565 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 4c0cd57525104ecdaed9a1dfc24d67ae [term 0 FOLLOWER]: Advancing to term 1
I20260620 11:42:49.043488 12565 raft_consensus.cc:504] T 00000000000000000000000000000000 P 4c0cd57525104ecdaed9a1dfc24d67ae [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4c0cd57525104ecdaed9a1dfc24d67ae" member_type: VOTER last_known_addr { host: "127.6.234.254" port: 36349 } }
I20260620 11:42:49.043594 12565 leader_election.cc:302] T 00000000000000000000000000000000 P 4c0cd57525104ecdaed9a1dfc24d67ae [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 4c0cd57525104ecdaed9a1dfc24d67ae; no voters:
I20260620 11:42:49.043754 12565 leader_election.cc:288] T 00000000000000000000000000000000 P 4c0cd57525104ecdaed9a1dfc24d67ae [CANDIDATE]: Term 1 election: Requested vote from peers
I20260620 11:42:49.043769 12568 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 11:42:49.037910 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53342 (local address 127.6.234.254:36349)
0620 11:42:49.038127 (+ 217us) server_negotiation.cc:206] Beginning negotiation
0620 11:42:49.038136 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0620 11:42:49.038168 (+ 32us) server_negotiation.cc:407] Connection header received
0620 11:42:49.038749 (+ 581us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 11:42:49.038758 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 11:42:49.038845 (+ 87us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 11:42:49.039001 (+ 156us) server_negotiation.cc:226] Negotiated authn=SASL
0620 11:42:49.039259 (+ 258us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 11:42:49.040625 (+ 1366us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 11:42:49.041429 (+ 804us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 11:42:49.042162 (+ 733us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 11:42:49.042283 (+ 121us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 11:42:49.043062 (+ 779us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 11:42:49.043066 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 11:42:49.043066 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 11:42:49.043271 (+ 205us) server_negotiation.cc:922] calling sasl_server_start()
0620 11:42:49.043312 (+ 41us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 11:42:49.043327 (+ 15us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 11:42:49.043398 (+ 71us) server_negotiation.cc:1035] Waiting for connection context
0620 11:42:49.043456 (+ 58us) server_negotiation.cc:299] Negotiation successful
0620 11:42:49.043540 (+ 84us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":138,"thread_start_us":77,"threads_started":1}
I20260620 11:42:49.043816 12570 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 4c0cd57525104ecdaed9a1dfc24d67ae [term 1 FOLLOWER]: Leader election won for term 1
I20260620 11:42:49.043977 12570 raft_consensus.cc:686] T 00000000000000000000000000000000 P 4c0cd57525104ecdaed9a1dfc24d67ae [term 1 LEADER]: Becoming Leader. State: Replica: 4c0cd57525104ecdaed9a1dfc24d67ae, State: Running, Role: LEADER
I20260620 11:42:49.044009 12565 sys_catalog.cc:564] T 00000000000000000000000000000000 P 4c0cd57525104ecdaed9a1dfc24d67ae [sys.catalog]: configured and running, proceeding with master startup.
I20260620 11:42:49.044056 12570 consensus_queue.cc:237] T 00000000000000000000000000000000 P 4c0cd57525104ecdaed9a1dfc24d67ae [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4c0cd57525104ecdaed9a1dfc24d67ae" member_type: VOTER last_known_addr { host: "127.6.234.254" port: 36349 } }
I20260620 11:42:49.044391 12572 sys_catalog.cc:455] T 00000000000000000000000000000000 P 4c0cd57525104ecdaed9a1dfc24d67ae [sys.catalog]: SysCatalogTable state changed. Reason: New leader 4c0cd57525104ecdaed9a1dfc24d67ae. Latest consensus state: current_term: 1 leader_uuid: "4c0cd57525104ecdaed9a1dfc24d67ae" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4c0cd57525104ecdaed9a1dfc24d67ae" member_type: VOTER last_known_addr { host: "127.6.234.254" port: 36349 } } }
I20260620 11:42:49.044471 12571 sys_catalog.cc:455] T 00000000000000000000000000000000 P 4c0cd57525104ecdaed9a1dfc24d67ae [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "4c0cd57525104ecdaed9a1dfc24d67ae" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4c0cd57525104ecdaed9a1dfc24d67ae" member_type: VOTER last_known_addr { host: "127.6.234.254" port: 36349 } } }
I20260620 11:42:49.044582 12571 sys_catalog.cc:458] T 00000000000000000000000000000000 P 4c0cd57525104ecdaed9a1dfc24d67ae [sys.catalog]: This master's current role is: LEADER
I20260620 11:42:49.044519 12572 sys_catalog.cc:458] T 00000000000000000000000000000000 P 4c0cd57525104ecdaed9a1dfc24d67ae [sys.catalog]: This master's current role is: LEADER
I20260620 11:42:49.044775 7083 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260620 11:42:49.045356 12579 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260620 11:42:49.045954 12579 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260620 11:42:49.047417 12579 catalog_manager.cc:1347] Generated new cluster ID: 3475de6477264faf9a41640d6c541411
I20260620 11:42:49.047472 12579 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260620 11:42:49.052660 12579 catalog_manager.cc:1370] Generated new certificate authority record
I20260620 11:42:49.053150 12579 catalog_manager.cc:1504] Loading token signing keys...
I20260620 11:42:49.063398 12579 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 4c0cd57525104ecdaed9a1dfc24d67ae: Generated new TSK 0
I20260620 11:42:49.063586 12579 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260620 11:42:49.118191 12568 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 11:42:49.075850 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53350 (local address 127.6.234.254:36349)
0620 11:42:49.075926 (+ 76us) server_negotiation.cc:206] Beginning negotiation
0620 11:42:49.075928 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 11:42:49.076033 (+ 105us) server_negotiation.cc:407] Connection header received
0620 11:42:49.076067 (+ 34us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 11:42:49.076068 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 11:42:49.076080 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 11:42:49.076146 (+ 66us) server_negotiation.cc:226] Negotiated authn=JWT
0620 11:42:49.076274 (+ 128us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 11:42:49.077066 (+ 792us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 11:42:49.077897 (+ 831us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 11:42:49.078224 (+ 327us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 11:42:49.078297 (+ 73us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 11:42:49.078441 (+ 144us) server_negotiation.cc:365] Received JWT_EXCHANGE NegotiatePB request
0620 11:42:49.117912 (+ 39471us) server_negotiation.cc:394] Sending RPC error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'default.url': curl error: Couldn't resolve host name: Could not resolve host: default.url
0620 11:42:49.118110 (+ 198us) negotiation.cc:327] Negotiation complete: Not authorized: Server connection negotiation failed: server connection from 127.0.0.1:53350: Failed to load JWKS: Error downloading JWKS from 'default.url': curl error: Couldn't resolve host name: Could not resolve host: default.url
Metrics: {"server-negotiator.queue_time_us":51}
W20260620 11:42:49.118278 12568 negotiation.cc:344] Unauthorized connection attempt: Server connection negotiation failed: server connection from 127.0.0.1:53350: Failed to load JWKS: Error downloading JWKS from 'default.url': curl error: Couldn't resolve host name: Could not resolve host: default.url
W20260620 11:42:49.118259 12594 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 11:42:49.075750 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.234.254:36349 (local address 127.0.0.1:53350)
0620 11:42:49.075922 (+ 172us) negotiation.cc:107] Waiting for socket to connect
0620 11:42:49.075940 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0620 11:42:49.075995 (+ 55us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 11:42:49.076151 (+ 156us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 11:42:49.076152 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 11:42:49.076154 (+ 2us) client_negotiation.cc:189] Negotiated authn=JWT
0620 11:42:49.076204 (+ 50us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 11:42:49.076205 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 11:42:49.077153 (+ 948us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 11:42:49.077154 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 11:42:49.077814 (+ 660us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 11:42:49.077817 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 11:42:49.078306 (+ 489us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 11:42:49.078307 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 11:42:49.078346 (+ 39us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 11:42:49.078351 (+ 5us) client_negotiation.cc:252] Sending JWT_EXCHANGE NegotiatePB request
0620 11:42:49.118055 (+ 39704us) client_negotiation.cc:283] Received error response from server: Runtime error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'default.url': curl error: Couldn't resolve host name: Could not resolve host: default.url
0620 11:42:49.118171 (+ 116us) negotiation.cc:327] Negotiation complete: Runtime error: Client connection negotiation failed: client connection to 127.6.234.254:36349: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'default.url': curl error: Couldn't resolve host name: Could not resolve host: default.url
Metrics: {"client-negotiator.queue_time_us":146,"thread_start_us":106,"threads_started":1}
I20260620 11:42:49.118477 7083 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task1JRg7N/build/release/bin/kudu with pid 12498
2026-06-20T11:42:49Z chronyd exiting
[ OK ] SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS (230 ms)
[ RUN ] SecurityITest.TestWorldReadableKeytab
[ OK ] SecurityITest.TestWorldReadableKeytab (106 ms)
[ RUN ] SecurityITest.TestWorldReadablePrivateKey
[ OK ] SecurityITest.TestWorldReadablePrivateKey (86 ms)
[ RUN ] SecurityITest.TestCorruptKerberosCC
Loading random data
Initializing database '/tmp/dist-test-task1JRg7N/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781955746769051-7083-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 20 11:42:49 dist-test-slave-cjc6 krb5kdc[12606](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 20 11:42:49 dist-test-slave-cjc6 krb5kdc[12606](info): set up 2 sockets
Jun 20 11:42:49 dist-test-slave-cjc6 krb5kdc[12606](info): commencing operation
krb5kdc: starting...
W20260620 11:42:50.341871 7083 mini_kdc.cc:121] Time spent starting KDC: real 1.013s user 0.000s sys 0.006s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task1JRg7N/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781955746769051-7083-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task1JRg7N/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781955746769051-7083-0/krb5kdc/test-user.keytab.
Jun 20 11:42:50 dist-test-slave-cjc6 krb5kdc[12606](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781955770, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-20T11:42:50Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-20T11:42:50Z Disabled control of system clock
WARNING: no policy specified for kudu/127.6.234.254@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.6.234.254@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.6.234.254 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task1JRg7N/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781955746769051-7083-0/krb5kdc/kudu_127.6.234.254.keytab.
Entry for principal kudu/127.6.234.254 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task1JRg7N/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781955746769051-7083-0/krb5kdc/kudu_127.6.234.254.keytab.
I20260620 11:42:50.454495 7083 external_mini_cluster.cc:1326] Running /tmp/dist-test-task1JRg7N/build/release/bin/kudu
/tmp/dist-test-task1JRg7N/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task1JRg7N/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781955746769051-7083-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task1JRg7N/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781955746769051-7083-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task1JRg7N/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781955746769051-7083-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task1JRg7N/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781955746769051-7083-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.6.234.254:42845
--webserver_interface=127.6.234.254
--webserver_port=0
--keytab_file=/tmp/dist-test-task1JRg7N/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781955746769051-7083-0/krb5kdc/kudu_127.6.234.254.keytab
--principal=kudu/127.6.234.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.6.234.212:38947
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.6.234.254:42845
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task1JRg7N/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781955746769051-7083-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task1JRg7N/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781955746769051-7083-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task1JRg7N/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781955746769051-7083-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 11:42:50.534533 12619 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 11:42:50.534780 12619 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 11:42:50.534819 12619 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 11:42:50.535976 12619 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 11:42:50.536046 12619 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 11:42:50.536069 12619 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 11:42:50.536090 12619 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 11:42:50.536120 12619 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 11:42:50.537554 12619 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.6.234.212:38947
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task1JRg7N/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781955746769051-7083-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task1JRg7N/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781955746769051-7083-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.6.234.254:42845
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task1JRg7N/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781955746769051-7083-0/krb5kdc/kudu_127.6.234.254.keytab
--principal=kudu/127.6.234.254
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.6.234.254:42845
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task1JRg7N/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781955746769051-7083-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.6.234.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.12619
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task1JRg7N/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781955746769051-7083-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 20 Jun 2026 10:32:37 UTC on ccycloud.quasar-bgyofv.root.comops.site
build id 4457173
I20260620 11:42:50.537853 12619 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 11:42:50.538074 12619 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 11:42:50.542986 12624 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 11:42:50.542941 12627 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 11:42:50.542989 12625 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 11:42:50.543304 12619 server_base.cc:1034] running on GCE node
I20260620 11:42:50.543530 12619 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 11:42:50.543830 12619 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 11:42:50.544983 12619 hybrid_clock.cc:648] HybridClock initialized: now 1781955770544961 us; error 35 us; skew 500 ppm
Jun 20 11:42:50 dist-test-slave-cjc6 krb5kdc[12606](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781955770, etypes {rep=17 tkt=17 ses=17}, kudu/127.6.234.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 11:42:50.547475 12619 init.cc:377] Logged in from keytab as kudu/127.6.234.254@KRBTEST.COM (short username kudu)
I20260620 11:42:50.547909 12619 webserver.cc:460] Webserver started at http://127.6.234.254:34027/ using document root <none> and password file <none>
I20260620 11:42:50.548132 12619 fs_manager.cc:359] Metadata directory not provided
I20260620 11:42:50.548199 12619 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 11:42:50.548347 12619 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 11:42:50.549398 12619 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task1JRg7N/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781955746769051-7083-0/minicluster-data/master-0/data/instance:
uuid: "6bc4ab00f5814429a09cb9fb582ebf94"
format_stamp: "Formatted at 2026-06-20 11:42:50 on dist-test-slave-cjc6"
server_key: "f441d7b0d6182eb975bf8ddcee3f00a2"
server_key_iv: "7c05cbc758baed3502e6851a09e36033"
server_key_version: "encryptionkey@0"
I20260620 11:42:50.549824 12619 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task1JRg7N/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781955746769051-7083-0/minicluster-data/master-0/wal/instance:
uuid: "6bc4ab00f5814429a09cb9fb582ebf94"
format_stamp: "Formatted at 2026-06-20 11:42:50 on dist-test-slave-cjc6"
server_key: "f441d7b0d6182eb975bf8ddcee3f00a2"
server_key_iv: "7c05cbc758baed3502e6851a09e36033"
server_key_version: "encryptionkey@0"
I20260620 11:42:50.551132 12619 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.001s sys 0.000s
I20260620 11:42:50.551862 12634 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 11:42:50.552032 12619 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260620 11:42:50.552129 12619 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task1JRg7N/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781955746769051-7083-0/minicluster-data/master-0/data,/tmp/dist-test-task1JRg7N/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781955746769051-7083-0/minicluster-data/master-0/wal
uuid: "6bc4ab00f5814429a09cb9fb582ebf94"
format_stamp: "Formatted at 2026-06-20 11:42:50 on dist-test-slave-cjc6"
server_key: "f441d7b0d6182eb975bf8ddcee3f00a2"
server_key_iv: "7c05cbc758baed3502e6851a09e36033"
server_key_version: "encryptionkey@0"
I20260620 11:42:50.552212 12619 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task1JRg7N/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781955746769051-7083-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task1JRg7N/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781955746769051-7083-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task1JRg7N/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781955746769051-7083-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 11:42:50.570986 12619 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 11:42:50.573195 12619 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 11:42:50.573355 12619 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 11:42:50.577159 12619 rpc_server.cc:307] RPC server started. Bound to: 127.6.234.254:42845
I20260620 11:42:50.577191 12686 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.6.234.254:42845 every 8 connection(s)
I20260620 11:42:50.577520 12619 server_base.cc:1166] Dumped server information to /tmp/dist-test-task1JRg7N/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781955746769051-7083-0/minicluster-data/master-0/data/info.pb
I20260620 11:42:50.578377 12687 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 11:42:50.578905 7083 external_mini_cluster.cc:1388] Started /tmp/dist-test-task1JRg7N/build/release/bin/kudu as pid 12619
I20260620 11:42:50.578987 7083 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task1JRg7N/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781955746769051-7083-0/minicluster-data/master-0/wal/instance
I20260620 11:42:50.579175 7083 external_mini_cluster.cc:1428] Setting key de6bfd9afc3204935f95a7f6c4152a88
I20260620 11:42:50.580919 12687 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 6bc4ab00f5814429a09cb9fb582ebf94: Bootstrap starting.
I20260620 11:42:50.581481 12687 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 6bc4ab00f5814429a09cb9fb582ebf94: Neither blocks nor log segments found. Creating new log.
I20260620 11:42:50.581807 12687 log.cc:824] T 00000000000000000000000000000000 P 6bc4ab00f5814429a09cb9fb582ebf94: Log is configured to *not* fsync() on all Append() calls
I20260620 11:42:50.582492 12687 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 6bc4ab00f5814429a09cb9fb582ebf94: No bootstrap required, opened a new log
I20260620 11:42:50.584400 12687 raft_consensus.cc:348] T 00000000000000000000000000000000 P 6bc4ab00f5814429a09cb9fb582ebf94 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6bc4ab00f5814429a09cb9fb582ebf94" member_type: VOTER last_known_addr { host: "127.6.234.254" port: 42845 } }
I20260620 11:42:50.584556 12687 raft_consensus.cc:374] T 00000000000000000000000000000000 P 6bc4ab00f5814429a09cb9fb582ebf94 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 11:42:50.584630 12687 raft_consensus.cc:729] T 00000000000000000000000000000000 P 6bc4ab00f5814429a09cb9fb582ebf94 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6bc4ab00f5814429a09cb9fb582ebf94, State: Initialized, Role: FOLLOWER
I20260620 11:42:50.584805 12687 consensus_queue.cc:260] T 00000000000000000000000000000000 P 6bc4ab00f5814429a09cb9fb582ebf94 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6bc4ab00f5814429a09cb9fb582ebf94" member_type: VOTER last_known_addr { host: "127.6.234.254" port: 42845 } }
I20260620 11:42:50.584900 12687 raft_consensus.cc:388] T 00000000000000000000000000000000 P 6bc4ab00f5814429a09cb9fb582ebf94 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260620 11:42:50.584944 12687 raft_consensus.cc:482] T 00000000000000000000000000000000 P 6bc4ab00f5814429a09cb9fb582ebf94 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260620 11:42:50.584990 12687 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 6bc4ab00f5814429a09cb9fb582ebf94 [term 0 FOLLOWER]: Advancing to term 1
Jun 20 11:42:50 dist-test-slave-cjc6 krb5kdc[12606](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781955770, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.6.234.254@KRBTEST.COM
I20260620 11:42:50.585551 12687 raft_consensus.cc:504] T 00000000000000000000000000000000 P 6bc4ab00f5814429a09cb9fb582ebf94 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6bc4ab00f5814429a09cb9fb582ebf94" member_type: VOTER last_known_addr { host: "127.6.234.254" port: 42845 } }
I20260620 11:42:50.585664 12687 leader_election.cc:302] T 00000000000000000000000000000000 P 6bc4ab00f5814429a09cb9fb582ebf94 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 6bc4ab00f5814429a09cb9fb582ebf94; no voters:
I20260620 11:42:50.585852 12687 leader_election.cc:288] T 00000000000000000000000000000000 P 6bc4ab00f5814429a09cb9fb582ebf94 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260620 11:42:50.585882 12692 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 6bc4ab00f5814429a09cb9fb582ebf94 [term 1 FOLLOWER]: Leader election won for term 1
I20260620 11:42:50.586057 12692 raft_consensus.cc:686] T 00000000000000000000000000000000 P 6bc4ab00f5814429a09cb9fb582ebf94 [term 1 LEADER]: Becoming Leader. State: Replica: 6bc4ab00f5814429a09cb9fb582ebf94, State: Running, Role: LEADER
I20260620 11:42:50.586184 12687 sys_catalog.cc:564] T 00000000000000000000000000000000 P 6bc4ab00f5814429a09cb9fb582ebf94 [sys.catalog]: configured and running, proceeding with master startup.
I20260620 11:42:50.586513 12692 consensus_queue.cc:237] T 00000000000000000000000000000000 P 6bc4ab00f5814429a09cb9fb582ebf94 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6bc4ab00f5814429a09cb9fb582ebf94" member_type: VOTER last_known_addr { host: "127.6.234.254" port: 42845 } }
I20260620 11:42:50.586865 12692 sys_catalog.cc:455] T 00000000000000000000000000000000 P 6bc4ab00f5814429a09cb9fb582ebf94 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 6bc4ab00f5814429a09cb9fb582ebf94. Latest consensus state: current_term: 1 leader_uuid: "6bc4ab00f5814429a09cb9fb582ebf94" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6bc4ab00f5814429a09cb9fb582ebf94" member_type: VOTER last_known_addr { host: "127.6.234.254" port: 42845 } } }
I20260620 11:42:50.586907 12693 sys_catalog.cc:455] T 00000000000000000000000000000000 P 6bc4ab00f5814429a09cb9fb582ebf94 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "6bc4ab00f5814429a09cb9fb582ebf94" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6bc4ab00f5814429a09cb9fb582ebf94" member_type: VOTER last_known_addr { host: "127.6.234.254" port: 42845 } } }
I20260620 11:42:50.587296 12693 sys_catalog.cc:458] T 00000000000000000000000000000000 P 6bc4ab00f5814429a09cb9fb582ebf94 [sys.catalog]: This master's current role is: LEADER
I20260620 11:42:50.587654 12704 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260620 11:42:50.588414 12704 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260620 11:42:50.587680 12692 sys_catalog.cc:458] T 00000000000000000000000000000000 P 6bc4ab00f5814429a09cb9fb582ebf94 [sys.catalog]: This master's current role is: LEADER
I20260620 11:42:50.590026 12690 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 11:42:50.579696 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60188 (local address 127.6.234.254:42845)
0620 11:42:50.579889 (+ 193us) server_negotiation.cc:206] Beginning negotiation
0620 11:42:50.579898 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0620 11:42:50.579931 (+ 33us) server_negotiation.cc:407] Connection header received
0620 11:42:50.580318 (+ 387us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 11:42:50.580324 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 11:42:50.580367 (+ 43us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 11:42:50.580511 (+ 144us) server_negotiation.cc:226] Negotiated authn=SASL
0620 11:42:50.581120 (+ 609us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 11:42:50.582170 (+ 1050us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 11:42:50.582947 (+ 777us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 11:42:50.583355 (+ 408us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 11:42:50.583468 (+ 113us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 11:42:50.585914 (+ 2446us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 11:42:50.585920 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 11:42:50.585921 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 11:42:50.585933 (+ 12us) server_negotiation.cc:922] calling sasl_server_start()
0620 11:42:50.588288 (+ 2355us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 11:42:50.588720 (+ 432us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 11:42:50.588722 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 11:42:50.588723 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 11:42:50.588821 (+ 98us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 11:42:50.589053 (+ 232us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 11:42:50.589054 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 11:42:50.589054 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 11:42:50.589322 (+ 268us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 11:42:50.589510 (+ 188us) server_negotiation.cc:1035] Waiting for connection context
0620 11:42:50.589585 (+ 75us) server_negotiation.cc:299] Negotiation successful
0620 11:42:50.589846 (+ 261us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":144,"thread_start_us":93,"threads_started":1}
I20260620 11:42:50.590368 12704 catalog_manager.cc:1347] Generated new cluster ID: 68004e6efc34466f83d496b0275f2d4f
I20260620 11:42:50.590431 12704 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260620 11:42:50.599918 12704 catalog_manager.cc:1370] Generated new certificate authority record
I20260620 11:42:50.600433 12704 catalog_manager.cc:1504] Loading token signing keys...
I20260620 11:42:50.602880 12704 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 6bc4ab00f5814429a09cb9fb582ebf94: Generated new TSK 0
I20260620 11:42:50.603035 12704 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.6.234.193@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.6.234.193@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.6.234.193 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task1JRg7N/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781955746769051-7083-0/krb5kdc/kudu_127.6.234.193.keytab.
Entry for principal kudu/127.6.234.193 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task1JRg7N/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781955746769051-7083-0/krb5kdc/kudu_127.6.234.193.keytab.
I20260620 11:42:50.630532 7083 external_mini_cluster.cc:1326] Running /tmp/dist-test-task1JRg7N/build/release/bin/kudu
/tmp/dist-test-task1JRg7N/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task1JRg7N/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781955746769051-7083-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-task1JRg7N/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781955746769051-7083-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task1JRg7N/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781955746769051-7083-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-task1JRg7N/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781955746769051-7083-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.6.234.193:0
--local_ip_for_outbound_sockets=127.6.234.193
--webserver_interface=127.6.234.193
--webserver_port=0
--tserver_master_addrs=127.6.234.254:42845
--keytab_file=/tmp/dist-test-task1JRg7N/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781955746769051-7083-0/krb5kdc/kudu_127.6.234.193.keytab
--principal=kudu/127.6.234.193
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.6.234.212:38947
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task1JRg7N/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781955746769051-7083-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task1JRg7N/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781955746769051-7083-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task1JRg7N/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781955746769051-7083-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 11:42:50.710227 12713 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 11:42:50.710434 12713 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 11:42:50.710458 12713 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 11:42:50.711612 12713 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 11:42:50.711654 12713 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 11:42:50.711680 12713 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.6.234.193
I20260620 11:42:50.713136 12713 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.6.234.212:38947
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task1JRg7N/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781955746769051-7083-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-task1JRg7N/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781955746769051-7083-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task1JRg7N/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781955746769051-7083-0/krb5kdc/kudu_127.6.234.193.keytab
--principal=kudu/127.6.234.193
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.6.234.193:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task1JRg7N/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781955746769051-7083-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.6.234.193
--webserver_port=0
--tserver_master_addrs=127.6.234.254:42845
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.12713
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.6.234.193
--log_dir=/tmp/dist-test-task1JRg7N/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781955746769051-7083-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 20 Jun 2026 10:32:37 UTC on ccycloud.quasar-bgyofv.root.comops.site
build id 4457173
I20260620 11:42:50.713372 12713 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 11:42:50.713614 12713 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 11:42:50.718421 12721 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 11:42:50.718433 12718 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 11:42:50.718639 12713 server_base.cc:1034] running on GCE node
W20260620 11:42:50.718411 12719 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 11:42:50.718887 12713 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 11:42:50.719147 12713 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 11:42:50.720310 12713 hybrid_clock.cc:648] HybridClock initialized: now 1781955770720296 us; error 27 us; skew 500 ppm
Jun 20 11:42:50 dist-test-slave-cjc6 krb5kdc[12606](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781955770, etypes {rep=17 tkt=17 ses=17}, kudu/127.6.234.193@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 11:42:50.722664 12713 init.cc:377] Logged in from keytab as kudu/127.6.234.193@KRBTEST.COM (short username kudu)
I20260620 11:42:50.723070 12713 webserver.cc:460] Webserver started at http://127.6.234.193:37397/ using document root <none> and password file <none>
I20260620 11:42:50.723286 12713 fs_manager.cc:359] Metadata directory not provided
I20260620 11:42:50.723352 12713 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 11:42:50.723500 12713 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 11:42:50.724519 12713 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task1JRg7N/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781955746769051-7083-0/minicluster-data/ts-0/data/instance:
uuid: "228eb5b97bed40e88f57524661028d18"
format_stamp: "Formatted at 2026-06-20 11:42:50 on dist-test-slave-cjc6"
server_key: "587e1571ecfaeb110bedc3a209f88a79"
server_key_iv: "e171f43ed8b6625f645cb8a32912d78c"
server_key_version: "encryptionkey@0"
I20260620 11:42:50.724870 12713 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task1JRg7N/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781955746769051-7083-0/minicluster-data/ts-0/wal/instance:
uuid: "228eb5b97bed40e88f57524661028d18"
format_stamp: "Formatted at 2026-06-20 11:42:50 on dist-test-slave-cjc6"
server_key: "587e1571ecfaeb110bedc3a209f88a79"
server_key_iv: "e171f43ed8b6625f645cb8a32912d78c"
server_key_version: "encryptionkey@0"
I20260620 11:42:50.726224 12713 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.001s sys 0.002s
I20260620 11:42:50.726907 12728 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 11:42:50.727068 12713 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260620 11:42:50.727166 12713 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task1JRg7N/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781955746769051-7083-0/minicluster-data/ts-0/data,/tmp/dist-test-task1JRg7N/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781955746769051-7083-0/minicluster-data/ts-0/wal
uuid: "228eb5b97bed40e88f57524661028d18"
format_stamp: "Formatted at 2026-06-20 11:42:50 on dist-test-slave-cjc6"
server_key: "587e1571ecfaeb110bedc3a209f88a79"
server_key_iv: "e171f43ed8b6625f645cb8a32912d78c"
server_key_version: "encryptionkey@0"
I20260620 11:42:50.727240 12713 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task1JRg7N/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781955746769051-7083-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-task1JRg7N/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781955746769051-7083-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-task1JRg7N/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781955746769051-7083-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 11:42:50.735773 12713 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 11:42:50.737473 12713 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 11:42:50.737620 12713 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 11:42:50.737855 12713 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 11:42:50.738143 12713 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 11:42:50.738195 12713 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 11:42:50.738235 12713 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 11:42:50.738263 12713 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 11:42:50.743922 12713 rpc_server.cc:307] RPC server started. Bound to: 127.6.234.193:39539
I20260620 11:42:50.743957 12841 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.6.234.193:39539 every 8 connection(s)
I20260620 11:42:50.744251 12713 server_base.cc:1166] Dumped server information to /tmp/dist-test-task1JRg7N/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781955746769051-7083-0/minicluster-data/ts-0/data/info.pb
I20260620 11:42:50.744935 7083 external_mini_cluster.cc:1388] Started /tmp/dist-test-task1JRg7N/build/release/bin/kudu as pid 12713
I20260620 11:42:50.745059 7083 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task1JRg7N/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781955746769051-7083-0/minicluster-data/ts-0/wal/instance
I20260620 11:42:50.745268 7083 external_mini_cluster.cc:1428] Setting key 72543f5bc6d0c13b21c7e98823d2a053
Jun 20 11:42:50 dist-test-slave-cjc6 krb5kdc[12606](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781955770, etypes {rep=17 tkt=17 ses=17}, kudu/127.6.234.193@KRBTEST.COM for kudu/127.6.234.254@KRBTEST.COM
WARNING: no policy specified for kudu/127.6.234.194@KRBTEST.COM; defaulting to no policy
I20260620 11:42:50.754510 12690 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 11:42:50.745072 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.6.234.193:59173 (local address 127.6.234.254:42845)
0620 11:42:50.745127 (+ 55us) server_negotiation.cc:206] Beginning negotiation
0620 11:42:50.745130 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0620 11:42:50.745372 (+ 242us) server_negotiation.cc:407] Connection header received
0620 11:42:50.746706 (+ 1334us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 11:42:50.746707 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 11:42:50.746716 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 11:42:50.746756 (+ 40us) server_negotiation.cc:226] Negotiated authn=SASL
0620 11:42:50.747585 (+ 829us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 11:42:50.748103 (+ 518us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 11:42:50.749108 (+ 1005us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 11:42:50.749329 (+ 221us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 11:42:50.749413 (+ 84us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 11:42:50.751532 (+ 2119us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 11:42:50.751533 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 11:42:50.751534 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 11:42:50.751542 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0620 11:42:50.753031 (+ 1489us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 11:42:50.753460 (+ 429us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 11:42:50.753461 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 11:42:50.753462 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 11:42:50.753524 (+ 62us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 11:42:50.753817 (+ 293us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 11:42:50.753818 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 11:42:50.753818 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 11:42:50.753984 (+ 166us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 11:42:50.754053 (+ 69us) server_negotiation.cc:1035] Waiting for connection context
0620 11:42:50.754327 (+ 274us) server_negotiation.cc:299] Negotiation successful
0620 11:42:50.754420 (+ 93us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":31}
I20260620 11:42:50.754681 12844 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 11:42:50.745032 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.234.254:42845 (local address 127.6.234.193:59173)
0620 11:42:50.745224 (+ 192us) negotiation.cc:107] Waiting for socket to connect
0620 11:42:50.745269 (+ 45us) client_negotiation.cc:174] Beginning negotiation
0620 11:42:50.746603 (+ 1334us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 11:42:50.746821 (+ 218us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 11:42:50.746825 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 11:42:50.747213 (+ 388us) client_negotiation.cc:189] Negotiated authn=SASL
0620 11:42:50.747499 (+ 286us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 11:42:50.747506 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 11:42:50.748201 (+ 695us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 11:42:50.748203 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 11:42:50.749004 (+ 801us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 11:42:50.749007 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 11:42:50.749422 (+ 415us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 11:42:50.749424 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 11:42:50.749555 (+ 131us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 11:42:50.750101 (+ 546us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 11:42:50.750109 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0620 11:42:50.751437 (+ 1328us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 11:42:50.753146 (+ 1709us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 11:42:50.753153 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 11:42:50.753155 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0620 11:42:50.753380 (+ 225us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 11:42:50.753600 (+ 220us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 11:42:50.753601 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 11:42:50.753601 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0620 11:42:50.753760 (+ 159us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 11:42:50.754043 (+ 283us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 11:42:50.754050 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 11:42:50.754191 (+ 141us) client_negotiation.cc:769] Sending connection context
0620 11:42:50.754302 (+ 111us) client_negotiation.cc:240] Negotiation successful
0620 11:42:50.754452 (+ 150us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":142,"thread_start_us":86,"threads_started":1}
I20260620 11:42:50.755148 12842 heartbeater.cc:347] Connected to a master server at 127.6.234.254:42845
I20260620 11:42:50.755229 12842 heartbeater.cc:464] Registering TS with master...
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.6.234.194@KRBTEST.COM" created.
I20260620 11:42:50.755411 12842 heartbeater.cc:511] Master 127.6.234.254:42845 requested a full tablet report, sending...
I20260620 11:42:50.755982 12651 ts_manager.cc:194] Registered new tserver with Master: 228eb5b97bed40e88f57524661028d18 (127.6.234.193:39539)
I20260620 11:42:50.756574 12651 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.6.234.193@KRBTEST.COM'} at 127.6.234.193:59173
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.6.234.194 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task1JRg7N/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781955746769051-7083-0/krb5kdc/kudu_127.6.234.194.keytab.
Entry for principal kudu/127.6.234.194 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task1JRg7N/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781955746769051-7083-0/krb5kdc/kudu_127.6.234.194.keytab.
I20260620 11:42:50.768751 7083 external_mini_cluster.cc:1326] Running /tmp/dist-test-task1JRg7N/build/release/bin/kudu
/tmp/dist-test-task1JRg7N/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task1JRg7N/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781955746769051-7083-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-task1JRg7N/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781955746769051-7083-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task1JRg7N/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781955746769051-7083-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-task1JRg7N/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781955746769051-7083-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.6.234.194:0
--local_ip_for_outbound_sockets=127.6.234.194
--webserver_interface=127.6.234.194
--webserver_port=0
--tserver_master_addrs=127.6.234.254:42845
--keytab_file=/tmp/dist-test-task1JRg7N/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781955746769051-7083-0/krb5kdc/kudu_127.6.234.194.keytab
--principal=kudu/127.6.234.194
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.6.234.212:38947
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task1JRg7N/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781955746769051-7083-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task1JRg7N/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781955746769051-7083-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task1JRg7N/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781955746769051-7083-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 11:42:50.848766 12847 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 11:42:50.849010 12847 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 11:42:50.849049 12847 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 11:42:50.850241 12847 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 11:42:50.850313 12847 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 11:42:50.850358 12847 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.6.234.194
I20260620 11:42:50.851814 12847 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.6.234.212:38947
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task1JRg7N/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781955746769051-7083-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-task1JRg7N/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781955746769051-7083-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task1JRg7N/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781955746769051-7083-0/krb5kdc/kudu_127.6.234.194.keytab
--principal=kudu/127.6.234.194
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.6.234.194:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task1JRg7N/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781955746769051-7083-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.6.234.194
--webserver_port=0
--tserver_master_addrs=127.6.234.254:42845
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.12847
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.6.234.194
--log_dir=/tmp/dist-test-task1JRg7N/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781955746769051-7083-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 20 Jun 2026 10:32:37 UTC on ccycloud.quasar-bgyofv.root.comops.site
build id 4457173
I20260620 11:42:50.852087 12847 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 11:42:50.852321 12847 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 11:42:50.857358 12853 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 11:42:50.857343 12855 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 11:42:50.857343 12852 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 11:42:50.857592 12847 server_base.cc:1034] running on GCE node
I20260620 11:42:50.857864 12847 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 11:42:50.858098 12847 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 11:42:50.859234 12847 hybrid_clock.cc:648] HybridClock initialized: now 1781955770859215 us; error 43 us; skew 500 ppm
Jun 20 11:42:50 dist-test-slave-cjc6 krb5kdc[12606](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781955770, etypes {rep=17 tkt=17 ses=17}, kudu/127.6.234.194@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 11:42:50.861639 12847 init.cc:377] Logged in from keytab as kudu/127.6.234.194@KRBTEST.COM (short username kudu)
I20260620 11:42:50.862073 12847 webserver.cc:460] Webserver started at http://127.6.234.194:40753/ using document root <none> and password file <none>
I20260620 11:42:50.862265 12847 fs_manager.cc:359] Metadata directory not provided
I20260620 11:42:50.862313 12847 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 11:42:50.862398 12847 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 11:42:50.863420 12847 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task1JRg7N/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781955746769051-7083-0/minicluster-data/ts-1/data/instance:
uuid: "dcc973f3a84e43da9d8797bf5ba54245"
format_stamp: "Formatted at 2026-06-20 11:42:50 on dist-test-slave-cjc6"
server_key: "6ddce1bc3f67036fa85b8959227f5d38"
server_key_iv: "929cf0a2fa9934393cf337ed7fbf2d11"
server_key_version: "encryptionkey@0"
I20260620 11:42:50.863704 12847 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task1JRg7N/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781955746769051-7083-0/minicluster-data/ts-1/wal/instance:
uuid: "dcc973f3a84e43da9d8797bf5ba54245"
format_stamp: "Formatted at 2026-06-20 11:42:50 on dist-test-slave-cjc6"
server_key: "6ddce1bc3f67036fa85b8959227f5d38"
server_key_iv: "929cf0a2fa9934393cf337ed7fbf2d11"
server_key_version: "encryptionkey@0"
I20260620 11:42:50.864961 12847 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260620 11:42:50.865655 12862 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 11:42:50.865854 12847 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260620 11:42:50.865924 12847 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task1JRg7N/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781955746769051-7083-0/minicluster-data/ts-1/data,/tmp/dist-test-task1JRg7N/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781955746769051-7083-0/minicluster-data/ts-1/wal
uuid: "dcc973f3a84e43da9d8797bf5ba54245"
format_stamp: "Formatted at 2026-06-20 11:42:50 on dist-test-slave-cjc6"
server_key: "6ddce1bc3f67036fa85b8959227f5d38"
server_key_iv: "929cf0a2fa9934393cf337ed7fbf2d11"
server_key_version: "encryptionkey@0"
I20260620 11:42:50.866015 12847 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task1JRg7N/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781955746769051-7083-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-task1JRg7N/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781955746769051-7083-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-task1JRg7N/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781955746769051-7083-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 11:42:50.886384 12847 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 11:42:50.888664 12847 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 11:42:50.888840 12847 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 11:42:50.889081 12847 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 11:42:50.889408 12847 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 11:42:50.889467 12847 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 11:42:50.889503 12847 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 11:42:50.889528 12847 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 11:42:50.895375 12847 rpc_server.cc:307] RPC server started. Bound to: 127.6.234.194:37239
I20260620 11:42:50.895426 12975 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.6.234.194:37239 every 8 connection(s)
I20260620 11:42:50.895738 12847 server_base.cc:1166] Dumped server information to /tmp/dist-test-task1JRg7N/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781955746769051-7083-0/minicluster-data/ts-1/data/info.pb
Jun 20 11:42:50 dist-test-slave-cjc6 krb5kdc[12606](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781955770, etypes {rep=17 tkt=17 ses=17}, kudu/127.6.234.194@KRBTEST.COM for kudu/127.6.234.254@KRBTEST.COM
I20260620 11:42:50.903347 7083 external_mini_cluster.cc:1388] Started /tmp/dist-test-task1JRg7N/build/release/bin/kudu as pid 12847
I20260620 11:42:50.903469 7083 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task1JRg7N/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781955746769051-7083-0/minicluster-data/ts-1/wal/instance
I20260620 11:42:50.903651 7083 external_mini_cluster.cc:1428] Setting key 47f6cb96154d29458271a37308557712
I20260620 11:42:50.905682 12690 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 11:42:50.896448 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.6.234.194:37533 (local address 127.6.234.254:42845)
0620 11:42:50.896520 (+ 72us) server_negotiation.cc:206] Beginning negotiation
0620 11:42:50.896522 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 11:42:50.896764 (+ 242us) server_negotiation.cc:407] Connection header received
0620 11:42:50.897793 (+ 1029us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 11:42:50.897795 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 11:42:50.897808 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 11:42:50.897896 (+ 88us) server_negotiation.cc:226] Negotiated authn=SASL
0620 11:42:50.898534 (+ 638us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 11:42:50.899055 (+ 521us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 11:42:50.900059 (+ 1004us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 11:42:50.900274 (+ 215us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 11:42:50.900367 (+ 93us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 11:42:50.902529 (+ 2162us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 11:42:50.902530 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 11:42:50.902531 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 11:42:50.902538 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0620 11:42:50.904168 (+ 1630us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 11:42:50.904735 (+ 567us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 11:42:50.904736 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 11:42:50.904736 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 11:42:50.904786 (+ 50us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 11:42:50.905072 (+ 286us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 11:42:50.905072 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 11:42:50.905072 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 11:42:50.905171 (+ 99us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 11:42:50.905221 (+ 50us) server_negotiation.cc:1035] Waiting for connection context
0620 11:42:50.905551 (+ 330us) server_negotiation.cc:299] Negotiation successful
0620 11:42:50.905619 (+ 68us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":46}
I20260620 11:42:50.906003 12978 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 11:42:50.896425 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.234.254:42845 (local address 127.6.234.194:37533)
0620 11:42:50.896620 (+ 195us) negotiation.cc:107] Waiting for socket to connect
0620 11:42:50.896656 (+ 36us) client_negotiation.cc:174] Beginning negotiation
0620 11:42:50.897631 (+ 975us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 11:42:50.897983 (+ 352us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 11:42:50.897987 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 11:42:50.898239 (+ 252us) client_negotiation.cc:189] Negotiated authn=SASL
0620 11:42:50.898431 (+ 192us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 11:42:50.898435 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 11:42:50.899153 (+ 718us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 11:42:50.899154 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 11:42:50.899964 (+ 810us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 11:42:50.899966 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 11:42:50.900376 (+ 410us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 11:42:50.900377 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 11:42:50.900504 (+ 127us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 11:42:50.900976 (+ 472us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 11:42:50.900983 (+ 7us) client_negotiation.cc:656] calling sasl_client_start()
0620 11:42:50.902414 (+ 1431us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 11:42:50.904244 (+ 1830us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 11:42:50.904249 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 11:42:50.904251 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0620 11:42:50.904642 (+ 391us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 11:42:50.904852 (+ 210us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 11:42:50.904853 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 11:42:50.904853 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0620 11:42:50.905007 (+ 154us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 11:42:50.905234 (+ 227us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 11:42:50.905239 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 11:42:50.905403 (+ 164us) client_negotiation.cc:769] Sending connection context
0620 11:42:50.905519 (+ 116us) client_negotiation.cc:240] Negotiation successful
0620 11:42:50.905713 (+ 194us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":158,"thread_start_us":96,"threads_started":1}
I20260620 11:42:50.906509 12976 heartbeater.cc:347] Connected to a master server at 127.6.234.254:42845
I20260620 11:42:50.906620 12976 heartbeater.cc:464] Registering TS with master...
I20260620 11:42:50.906793 12976 heartbeater.cc:511] Master 127.6.234.254:42845 requested a full tablet report, sending...
I20260620 11:42:50.907195 12651 ts_manager.cc:194] Registered new tserver with Master: dcc973f3a84e43da9d8797bf5ba54245 (127.6.234.194:37239)
I20260620 11:42:50.907577 12651 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.6.234.194@KRBTEST.COM'} at 127.6.234.194:37533
WARNING: no policy specified for kudu/127.6.234.195@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.6.234.195@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.6.234.195 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task1JRg7N/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781955746769051-7083-0/krb5kdc/kudu_127.6.234.195.keytab.
Entry for principal kudu/127.6.234.195 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task1JRg7N/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781955746769051-7083-0/krb5kdc/kudu_127.6.234.195.keytab.
I20260620 11:42:50.926720 7083 external_mini_cluster.cc:1326] Running /tmp/dist-test-task1JRg7N/build/release/bin/kudu
/tmp/dist-test-task1JRg7N/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task1JRg7N/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781955746769051-7083-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-task1JRg7N/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781955746769051-7083-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task1JRg7N/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781955746769051-7083-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-task1JRg7N/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781955746769051-7083-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.6.234.195:0
--local_ip_for_outbound_sockets=127.6.234.195
--webserver_interface=127.6.234.195
--webserver_port=0
--tserver_master_addrs=127.6.234.254:42845
--keytab_file=/tmp/dist-test-task1JRg7N/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781955746769051-7083-0/krb5kdc/kudu_127.6.234.195.keytab
--principal=kudu/127.6.234.195
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.6.234.212:38947
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task1JRg7N/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781955746769051-7083-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task1JRg7N/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781955746769051-7083-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task1JRg7N/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781955746769051-7083-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 11:42:51.006455 12981 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 11:42:51.006666 12981 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 11:42:51.006690 12981 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 11:42:51.007790 12981 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 11:42:51.007826 12981 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 11:42:51.007851 12981 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.6.234.195
I20260620 11:42:51.009244 12981 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.6.234.212:38947
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task1JRg7N/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781955746769051-7083-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-task1JRg7N/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781955746769051-7083-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task1JRg7N/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781955746769051-7083-0/krb5kdc/kudu_127.6.234.195.keytab
--principal=kudu/127.6.234.195
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.6.234.195:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task1JRg7N/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781955746769051-7083-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.6.234.195
--webserver_port=0
--tserver_master_addrs=127.6.234.254:42845
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.12981
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.6.234.195
--log_dir=/tmp/dist-test-task1JRg7N/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781955746769051-7083-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 20 Jun 2026 10:32:37 UTC on ccycloud.quasar-bgyofv.root.comops.site
build id 4457173
I20260620 11:42:51.009429 12981 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 11:42:51.009600 12981 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 11:42:51.014580 12987 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 11:42:51.015864 12981 server_base.cc:1034] running on GCE node
W20260620 11:42:51.016075 12986 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 11:42:51.016458 12989 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 11:42:51.016698 12981 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 11:42:51.016953 12981 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 11:42:51.018115 12981 hybrid_clock.cc:648] HybridClock initialized: now 1781955771018105 us; error 46 us; skew 500 ppm
Jun 20 11:42:51 dist-test-slave-cjc6 krb5kdc[12606](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781955771, etypes {rep=17 tkt=17 ses=17}, kudu/127.6.234.195@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 11:42:51.020701 12981 init.cc:377] Logged in from keytab as kudu/127.6.234.195@KRBTEST.COM (short username kudu)
I20260620 11:42:51.021142 12981 webserver.cc:460] Webserver started at http://127.6.234.195:41927/ using document root <none> and password file <none>
I20260620 11:42:51.021379 12981 fs_manager.cc:359] Metadata directory not provided
I20260620 11:42:51.021447 12981 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 11:42:51.021581 12981 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 11:42:51.022689 12981 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task1JRg7N/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781955746769051-7083-0/minicluster-data/ts-2/data/instance:
uuid: "b43b107f354947bab87a124d1b30b4fc"
format_stamp: "Formatted at 2026-06-20 11:42:51 on dist-test-slave-cjc6"
server_key: "ea5fe41a8bb4c35c5937b407d0f6496c"
server_key_iv: "5a42157db7dc1741a102e1b85804aed6"
server_key_version: "encryptionkey@0"
I20260620 11:42:51.023062 12981 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task1JRg7N/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781955746769051-7083-0/minicluster-data/ts-2/wal/instance:
uuid: "b43b107f354947bab87a124d1b30b4fc"
format_stamp: "Formatted at 2026-06-20 11:42:51 on dist-test-slave-cjc6"
server_key: "ea5fe41a8bb4c35c5937b407d0f6496c"
server_key_iv: "5a42157db7dc1741a102e1b85804aed6"
server_key_version: "encryptionkey@0"
I20260620 11:42:51.024351 12981 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260620 11:42:51.025125 12996 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 11:42:51.025319 12981 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260620 11:42:51.025430 12981 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task1JRg7N/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781955746769051-7083-0/minicluster-data/ts-2/data,/tmp/dist-test-task1JRg7N/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781955746769051-7083-0/minicluster-data/ts-2/wal
uuid: "b43b107f354947bab87a124d1b30b4fc"
format_stamp: "Formatted at 2026-06-20 11:42:51 on dist-test-slave-cjc6"
server_key: "ea5fe41a8bb4c35c5937b407d0f6496c"
server_key_iv: "5a42157db7dc1741a102e1b85804aed6"
server_key_version: "encryptionkey@0"
I20260620 11:42:51.025519 12981 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task1JRg7N/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781955746769051-7083-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-task1JRg7N/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781955746769051-7083-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-task1JRg7N/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781955746769051-7083-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 11:42:51.033021 12981 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 11:42:51.034806 12981 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 11:42:51.034965 12981 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 11:42:51.035182 12981 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 11:42:51.035488 12981 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 11:42:51.035545 12981 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 11:42:51.035583 12981 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 11:42:51.035611 12981 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 11:42:51.041159 12981 rpc_server.cc:307] RPC server started. Bound to: 127.6.234.195:42239
I20260620 11:42:51.041224 13109 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.6.234.195:42239 every 8 connection(s)
I20260620 11:42:51.041543 12981 server_base.cc:1166] Dumped server information to /tmp/dist-test-task1JRg7N/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781955746769051-7083-0/minicluster-data/ts-2/data/info.pb
Jun 20 11:42:51 dist-test-slave-cjc6 krb5kdc[12606](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781955771, etypes {rep=17 tkt=17 ses=17}, kudu/127.6.234.195@KRBTEST.COM for kudu/127.6.234.254@KRBTEST.COM
I20260620 11:42:51.050460 12690 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 11:42:51.042259 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.6.234.195:35403 (local address 127.6.234.254:42845)
0620 11:42:51.042337 (+ 78us) server_negotiation.cc:206] Beginning negotiation
0620 11:42:51.042340 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0620 11:42:51.042512 (+ 172us) server_negotiation.cc:407] Connection header received
0620 11:42:51.043502 (+ 990us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 11:42:51.043503 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 11:42:51.043514 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 11:42:51.043561 (+ 47us) server_negotiation.cc:226] Negotiated authn=SASL
0620 11:42:51.044128 (+ 567us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 11:42:51.044618 (+ 490us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 11:42:51.045538 (+ 920us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 11:42:51.045798 (+ 260us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 11:42:51.045858 (+ 60us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 11:42:51.047884 (+ 2026us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 11:42:51.047885 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 11:42:51.047885 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 11:42:51.047892 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0620 11:42:51.049095 (+ 1203us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 11:42:51.049452 (+ 357us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 11:42:51.049453 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 11:42:51.049453 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 11:42:51.049501 (+ 48us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 11:42:51.049806 (+ 305us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 11:42:51.049807 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 11:42:51.049807 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 11:42:51.049912 (+ 105us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 11:42:51.049968 (+ 56us) server_negotiation.cc:1035] Waiting for connection context
0620 11:42:51.050304 (+ 336us) server_negotiation.cc:299] Negotiation successful
0620 11:42:51.050383 (+ 79us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":46}
I20260620 11:42:51.050539 13112 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 11:42:51.042221 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.234.254:42845 (local address 127.6.234.195:35403)
0620 11:42:51.042400 (+ 179us) negotiation.cc:107] Waiting for socket to connect
0620 11:42:51.042433 (+ 33us) client_negotiation.cc:174] Beginning negotiation
0620 11:42:51.043403 (+ 970us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 11:42:51.043620 (+ 217us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 11:42:51.043624 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 11:42:51.043872 (+ 248us) client_negotiation.cc:189] Negotiated authn=SASL
0620 11:42:51.044051 (+ 179us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 11:42:51.044055 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 11:42:51.044695 (+ 640us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 11:42:51.044696 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 11:42:51.045464 (+ 768us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 11:42:51.045466 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 11:42:51.045884 (+ 418us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 11:42:51.045885 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 11:42:51.045987 (+ 102us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 11:42:51.046448 (+ 461us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 11:42:51.046456 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0620 11:42:51.047804 (+ 1348us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 11:42:51.049172 (+ 1368us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 11:42:51.049177 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 11:42:51.049178 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0620 11:42:51.049389 (+ 211us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 11:42:51.049559 (+ 170us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 11:42:51.049559 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 11:42:51.049560 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0620 11:42:51.049651 (+ 91us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 11:42:51.049995 (+ 344us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 11:42:51.050000 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 11:42:51.050114 (+ 114us) client_negotiation.cc:769] Sending connection context
0620 11:42:51.050253 (+ 139us) client_negotiation.cc:240] Negotiation successful
0620 11:42:51.050382 (+ 129us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":137,"thread_start_us":97,"threads_started":1}
I20260620 11:42:51.051012 7083 external_mini_cluster.cc:1388] Started /tmp/dist-test-task1JRg7N/build/release/bin/kudu as pid 12981
I20260620 11:42:51.051117 7083 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task1JRg7N/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781955746769051-7083-0/minicluster-data/ts-2/wal/instance
I20260620 11:42:51.051183 13110 heartbeater.cc:347] Connected to a master server at 127.6.234.254:42845
I20260620 11:42:51.051316 13110 heartbeater.cc:464] Registering TS with master...
I20260620 11:42:51.051332 7083 external_mini_cluster.cc:1428] Setting key c075ce30a19ee976731d9e2dfadc6346
I20260620 11:42:51.051494 13110 heartbeater.cc:511] Master 127.6.234.254:42845 requested a full tablet report, sending...
I20260620 11:42:51.051877 12651 ts_manager.cc:194] Registered new tserver with Master: b43b107f354947bab87a124d1b30b4fc (127.6.234.195:42239)
I20260620 11:42:51.052300 7083 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260620 11:42:51.052322 12651 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.6.234.195@KRBTEST.COM'} at 127.6.234.195:35403
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal test-admin with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task1JRg7N/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781955746769051-7083-0/krb5kdc/test-admin.keytab.
Entry for principal test-admin with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task1JRg7N/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781955746769051-7083-0/krb5kdc/test-admin.keytab.
Jun 20 11:42:51 dist-test-slave-cjc6 krb5kdc[12606](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781955771, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 11:42:51.067152 7083 init.cc:377] Logged in from keytab as test-admin@KRBTEST.COM (short username test-admin)
W20260620 11:42:51.068969 13117 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 11:42:51.068035 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53682 (local address 127.6.234.193:39539)
0620 11:42:51.068249 (+ 214us) server_negotiation.cc:206] Beginning negotiation
0620 11:42:51.068254 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 11:42:51.068274 (+ 20us) server_negotiation.cc:407] Connection header received
0620 11:42:51.068299 (+ 25us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 11:42:51.068301 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 11:42:51.068314 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 11:42:51.068380 (+ 66us) server_negotiation.cc:226] Negotiated authn=SASL
0620 11:42:51.068854 (+ 474us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:53682: BlockingRecv error: recv got EOF from 127.0.0.1:53682 (error 108)
Metrics: {"server-negotiator.queue_time_us":153,"thread_start_us":104,"threads_started":1}
W20260620 11:42:51.069010 7083 init.cc:300] Could not find kerberos principal in credential cache '/tmp/dist-test-task1JRg7N/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781955746769051-7083-0/krb5kdc/krb5cc' of type FILE
Jun 20 11:42:51 dist-test-slave-cjc6 krb5kdc[12606](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781955771, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 11:42:51.070148 7083 init.cc:377] Logged in from keytab as test-admin@KRBTEST.COM (short username test-admin)
Jun 20 11:42:51 dist-test-slave-cjc6 krb5kdc[12606](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781955771, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 11:42:51.071115 7083 init.cc:295] Successfully reacquired a new kerberos TGT
Jun 20 11:42:51 dist-test-slave-cjc6 krb5kdc[12606](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781955771, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.6.234.193@KRBTEST.COM
I20260620 11:42:51.078971 13117 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 11:42:51.071703 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53698 (local address 127.6.234.193:39539)
0620 11:42:51.071763 (+ 60us) server_negotiation.cc:206] Beginning negotiation
0620 11:42:51.071765 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 11:42:51.071809 (+ 44us) server_negotiation.cc:407] Connection header received
0620 11:42:51.071851 (+ 42us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 11:42:51.071852 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 11:42:51.071862 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 11:42:51.071910 (+ 48us) server_negotiation.cc:226] Negotiated authn=SASL
0620 11:42:51.072294 (+ 384us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 11:42:51.073017 (+ 723us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 11:42:51.073778 (+ 761us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 11:42:51.074095 (+ 317us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 11:42:51.074153 (+ 58us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 11:42:51.076363 (+ 2210us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 11:42:51.076369 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 11:42:51.076370 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 11:42:51.076377 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0620 11:42:51.077869 (+ 1492us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 11:42:51.078183 (+ 314us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 11:42:51.078185 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 11:42:51.078186 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 11:42:51.078239 (+ 53us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 11:42:51.078404 (+ 165us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 11:42:51.078404 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 11:42:51.078405 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 11:42:51.078522 (+ 117us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 11:42:51.078618 (+ 96us) server_negotiation.cc:1035] Waiting for connection context
0620 11:42:51.078750 (+ 132us) server_negotiation.cc:299] Negotiation successful
0620 11:42:51.078811 (+ 61us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":42}
W20260620 11:42:51.080685 13117 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 11:42:51.080127 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53702 (local address 127.6.234.193:39539)
0620 11:42:51.080160 (+ 33us) server_negotiation.cc:206] Beginning negotiation
0620 11:42:51.080162 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 11:42:51.080231 (+ 69us) server_negotiation.cc:407] Connection header received
0620 11:42:51.080300 (+ 69us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 11:42:51.080301 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 11:42:51.080311 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 11:42:51.080362 (+ 51us) server_negotiation.cc:226] Negotiated authn=SASL
0620 11:42:51.080637 (+ 275us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:53702: BlockingRecv error: recv got EOF from 127.0.0.1:53702 (error 108)
Metrics: {"server-negotiator.queue_time_us":16}
W20260620 11:42:51.080855 7083 init.cc:300] Could not find kerberos principal in credential cache '/tmp/dist-test-task1JRg7N/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781955746769051-7083-0/krb5kdc/krb5cc' of type FILE
Jun 20 11:42:51 dist-test-slave-cjc6 krb5kdc[12606](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781955771, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 11:42:51.081796 7083 init.cc:377] Logged in from keytab as test-admin@KRBTEST.COM (short username test-admin)
Jun 20 11:42:51 dist-test-slave-cjc6 krb5kdc[12606](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781955771, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 11:42:51.082654 7083 init.cc:295] Successfully reacquired a new kerberos TGT
Jun 20 11:42:51 dist-test-slave-cjc6 krb5kdc[12606](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781955771, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.6.234.193@KRBTEST.COM
I20260620 11:42:51.089501 13117 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 11:42:51.083104 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53708 (local address 127.6.234.193:39539)
0620 11:42:51.083136 (+ 32us) server_negotiation.cc:206] Beginning negotiation
0620 11:42:51.083138 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 11:42:51.083175 (+ 37us) server_negotiation.cc:407] Connection header received
0620 11:42:51.083217 (+ 42us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 11:42:51.083218 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 11:42:51.083224 (+ 6us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 11:42:51.083270 (+ 46us) server_negotiation.cc:226] Negotiated authn=SASL
0620 11:42:51.083588 (+ 318us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 11:42:51.084062 (+ 474us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 11:42:51.084729 (+ 667us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 11:42:51.084932 (+ 203us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 11:42:51.084995 (+ 63us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 11:42:51.087101 (+ 2106us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 11:42:51.087103 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 11:42:51.087103 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 11:42:51.087110 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0620 11:42:51.088426 (+ 1316us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 11:42:51.088780 (+ 354us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 11:42:51.088781 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 11:42:51.088782 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 11:42:51.088834 (+ 52us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 11:42:51.089036 (+ 202us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 11:42:51.089037 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 11:42:51.089037 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 11:42:51.089129 (+ 92us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 11:42:51.089228 (+ 99us) server_negotiation.cc:1035] Waiting for connection context
0620 11:42:51.089333 (+ 105us) server_negotiation.cc:299] Negotiation successful
0620 11:42:51.089407 (+ 74us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":16}
W20260620 11:42:51.090919 13117 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 11:42:51.090419 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53718 (local address 127.6.234.193:39539)
0620 11:42:51.090449 (+ 30us) server_negotiation.cc:206] Beginning negotiation
0620 11:42:51.090451 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 11:42:51.090504 (+ 53us) server_negotiation.cc:407] Connection header received
0620 11:42:51.090546 (+ 42us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 11:42:51.090547 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 11:42:51.090553 (+ 6us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 11:42:51.090594 (+ 41us) server_negotiation.cc:226] Negotiated authn=SASL
0620 11:42:51.090869 (+ 275us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:53718: BlockingRecv error: recv got EOF from 127.0.0.1:53718 (error 108)
Metrics: {"server-negotiator.queue_time_us":15}
W20260620 11:42:51.091012 7083 init.cc:300] Could not find kerberos principal in credential cache '/tmp/dist-test-task1JRg7N/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781955746769051-7083-0/krb5kdc/krb5cc' of type FILE
Jun 20 11:42:51 dist-test-slave-cjc6 krb5kdc[12606](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781955771, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 11:42:51.091900 7083 init.cc:377] Logged in from keytab as test-admin@KRBTEST.COM (short username test-admin)
Jun 20 11:42:51 dist-test-slave-cjc6 krb5kdc[12606](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781955771, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 11:42:51.092682 7083 init.cc:295] Successfully reacquired a new kerberos TGT
Jun 20 11:42:51 dist-test-slave-cjc6 krb5kdc[12606](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781955771, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.6.234.193@KRBTEST.COM
I20260620 11:42:51.099548 13117 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 11:42:51.093131 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53726 (local address 127.6.234.193:39539)
0620 11:42:51.093166 (+ 35us) server_negotiation.cc:206] Beginning negotiation
0620 11:42:51.093168 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 11:42:51.093275 (+ 107us) server_negotiation.cc:407] Connection header received
0620 11:42:51.093352 (+ 77us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 11:42:51.093353 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 11:42:51.093362 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 11:42:51.093434 (+ 72us) server_negotiation.cc:226] Negotiated authn=SASL
0620 11:42:51.093853 (+ 419us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 11:42:51.094300 (+ 447us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 11:42:51.094987 (+ 687us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 11:42:51.095176 (+ 189us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 11:42:51.095254 (+ 78us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 11:42:51.097081 (+ 1827us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 11:42:51.097082 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 11:42:51.097082 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 11:42:51.097089 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0620 11:42:51.098271 (+ 1182us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 11:42:51.098612 (+ 341us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 11:42:51.098613 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 11:42:51.098613 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 11:42:51.098664 (+ 51us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 11:42:51.098938 (+ 274us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 11:42:51.098938 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 11:42:51.098939 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 11:42:51.099104 (+ 165us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 11:42:51.099220 (+ 116us) server_negotiation.cc:1035] Waiting for connection context
0620 11:42:51.099344 (+ 124us) server_negotiation.cc:299] Negotiation successful
0620 11:42:51.099437 (+ 93us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":18}
I20260620 11:42:51.102274 7083 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task1JRg7N/build/release/bin/kudu with pid 12713
I20260620 11:42:51.108930 7083 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task1JRg7N/build/release/bin/kudu with pid 12847
I20260620 11:42:51.113574 7083 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task1JRg7N/build/release/bin/kudu with pid 12981
I20260620 11:42:51.118016 7083 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task1JRg7N/build/release/bin/kudu with pid 12619
2026-06-20T11:42:51Z chronyd exiting
[ OK ] SecurityITest.TestCorruptKerberosCC (1809 ms)
[ RUN ] SecurityITest.TestNonDefaultPrincipalMultipleMaster
Loading random data
Initializing database '/tmp/dist-test-task1JRg7N/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781955746769051-7083-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 20 11:42:51 dist-test-slave-cjc6 krb5kdc[13130](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 20 11:42:51 dist-test-slave-cjc6 krb5kdc[13130](info): set up 2 sockets
Jun 20 11:42:51 dist-test-slave-cjc6 krb5kdc[13130](info): commencing operation
krb5kdc: starting...
W20260620 11:42:52.154070 7083 mini_kdc.cc:121] Time spent starting KDC: real 1.016s user 0.001s sys 0.004s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task1JRg7N/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781955746769051-7083-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task1JRg7N/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781955746769051-7083-0/krb5kdc/test-user.keytab.
Jun 20 11:42:52 dist-test-slave-cjc6 krb5kdc[13130](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781955772, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-20T11:42:52Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-20T11:42:52Z Disabled control of system clock
WARNING: no policy specified for oryx/127.6.234.254@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.6.234.254@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.6.234.254 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task1JRg7N/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781955746769051-7083-0/krb5kdc/oryx_127.6.234.254.keytab.
Entry for principal oryx/127.6.234.254 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task1JRg7N/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781955746769051-7083-0/krb5kdc/oryx_127.6.234.254.keytab.
I20260620 11:42:52.266360 7083 external_mini_cluster.cc:1326] Running /tmp/dist-test-task1JRg7N/build/release/bin/kudu
/tmp/dist-test-task1JRg7N/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task1JRg7N/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781955746769051-7083-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task1JRg7N/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781955746769051-7083-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task1JRg7N/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781955746769051-7083-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task1JRg7N/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781955746769051-7083-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.6.234.254:39029
--webserver_interface=127.6.234.254
--webserver_port=0
--keytab_file=/tmp/dist-test-task1JRg7N/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781955746769051-7083-0/krb5kdc/oryx_127.6.234.254.keytab
--principal=oryx/127.6.234.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.6.234.212:38681
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.6.234.254:39029
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task1JRg7N/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781955746769051-7083-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task1JRg7N/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781955746769051-7083-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task1JRg7N/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781955746769051-7083-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 11:42:52.345216 13143 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 11:42:52.345470 13143 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 11:42:52.345508 13143 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 11:42:52.346760 13143 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 11:42:52.346824 13143 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 11:42:52.346848 13143 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 11:42:52.346874 13143 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 11:42:52.346905 13143 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 11:42:52.348342 13143 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.6.234.212:38681
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task1JRg7N/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781955746769051-7083-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task1JRg7N/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781955746769051-7083-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.6.234.254:39029
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task1JRg7N/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781955746769051-7083-0/krb5kdc/oryx_127.6.234.254.keytab
--principal=oryx/127.6.234.254
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.6.234.254:39029
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task1JRg7N/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781955746769051-7083-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.6.234.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.13143
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task1JRg7N/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781955746769051-7083-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 20 Jun 2026 10:32:37 UTC on ccycloud.quasar-bgyofv.root.comops.site
build id 4457173
I20260620 11:42:52.348628 13143 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 11:42:52.348848 13143 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 11:42:52.353660 13149 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 11:42:52.353660 13148 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 11:42:52.353695 13151 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 11:42:52.353745 13143 server_base.cc:1034] running on GCE node
I20260620 11:42:52.354125 13143 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 11:42:52.354413 13143 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 11:42:52.355557 13143 hybrid_clock.cc:648] HybridClock initialized: now 1781955772355531 us; error 46 us; skew 500 ppm
Jun 20 11:42:52 dist-test-slave-cjc6 krb5kdc[13130](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781955772, etypes {rep=17 tkt=17 ses=17}, oryx/127.6.234.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 11:42:52.357969 13143 init.cc:377] Logged in from keytab as oryx/127.6.234.254@KRBTEST.COM (short username oryx)
I20260620 11:42:52.358383 13143 webserver.cc:460] Webserver started at http://127.6.234.254:40905/ using document root <none> and password file <none>
I20260620 11:42:52.358616 13143 fs_manager.cc:359] Metadata directory not provided
I20260620 11:42:52.358696 13143 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 11:42:52.358851 13143 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 11:42:52.359879 13143 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task1JRg7N/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781955746769051-7083-0/minicluster-data/master-0/data/instance:
uuid: "cc8526b37e1a4e17aa4785a4a8f15156"
format_stamp: "Formatted at 2026-06-20 11:42:52 on dist-test-slave-cjc6"
server_key: "fa8914bf6077345ec940db45b2b5bcae"
server_key_iv: "f4378386d98a0e968884792ebbe65ca9"
server_key_version: "encryptionkey@0"
I20260620 11:42:52.360231 13143 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task1JRg7N/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781955746769051-7083-0/minicluster-data/master-0/wal/instance:
uuid: "cc8526b37e1a4e17aa4785a4a8f15156"
format_stamp: "Formatted at 2026-06-20 11:42:52 on dist-test-slave-cjc6"
server_key: "fa8914bf6077345ec940db45b2b5bcae"
server_key_iv: "f4378386d98a0e968884792ebbe65ca9"
server_key_version: "encryptionkey@0"
I20260620 11:42:52.361510 13143 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.001s
I20260620 11:42:52.362327 13158 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 11:42:52.362499 13143 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.000s
I20260620 11:42:52.362591 13143 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task1JRg7N/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781955746769051-7083-0/minicluster-data/master-0/data,/tmp/dist-test-task1JRg7N/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781955746769051-7083-0/minicluster-data/master-0/wal
uuid: "cc8526b37e1a4e17aa4785a4a8f15156"
format_stamp: "Formatted at 2026-06-20 11:42:52 on dist-test-slave-cjc6"
server_key: "fa8914bf6077345ec940db45b2b5bcae"
server_key_iv: "f4378386d98a0e968884792ebbe65ca9"
server_key_version: "encryptionkey@0"
I20260620 11:42:52.362677 13143 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task1JRg7N/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781955746769051-7083-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task1JRg7N/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781955746769051-7083-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task1JRg7N/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781955746769051-7083-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 11:42:52.378775 13143 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 11:42:52.380954 13143 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 11:42:52.381120 13143 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 11:42:52.385152 13143 rpc_server.cc:307] RPC server started. Bound to: 127.6.234.254:39029
I20260620 11:42:52.385202 13210 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.6.234.254:39029 every 8 connection(s)
I20260620 11:42:52.385562 13143 server_base.cc:1166] Dumped server information to /tmp/dist-test-task1JRg7N/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781955746769051-7083-0/minicluster-data/master-0/data/info.pb
I20260620 11:42:52.386612 13211 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 11:42:52.388844 13211 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P cc8526b37e1a4e17aa4785a4a8f15156: Bootstrap starting.
I20260620 11:42:52.389424 13211 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P cc8526b37e1a4e17aa4785a4a8f15156: Neither blocks nor log segments found. Creating new log.
I20260620 11:42:52.389737 13211 log.cc:824] T 00000000000000000000000000000000 P cc8526b37e1a4e17aa4785a4a8f15156: Log is configured to *not* fsync() on all Append() calls
I20260620 11:42:52.390405 13211 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P cc8526b37e1a4e17aa4785a4a8f15156: No bootstrap required, opened a new log
I20260620 11:42:52.390663 7083 external_mini_cluster.cc:1388] Started /tmp/dist-test-task1JRg7N/build/release/bin/kudu as pid 13143
I20260620 11:42:52.390774 7083 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task1JRg7N/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781955746769051-7083-0/minicluster-data/master-0/wal/instance
I20260620 11:42:52.390961 7083 external_mini_cluster.cc:1428] Setting key d0a33e954a5d1e74e36af16f989f9684
I20260620 11:42:52.393006 13211 raft_consensus.cc:348] T 00000000000000000000000000000000 P cc8526b37e1a4e17aa4785a4a8f15156 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "cc8526b37e1a4e17aa4785a4a8f15156" member_type: VOTER last_known_addr { host: "127.6.234.254" port: 39029 } }
I20260620 11:42:52.393184 13211 raft_consensus.cc:374] T 00000000000000000000000000000000 P cc8526b37e1a4e17aa4785a4a8f15156 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 11:42:52.393229 13211 raft_consensus.cc:729] T 00000000000000000000000000000000 P cc8526b37e1a4e17aa4785a4a8f15156 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: cc8526b37e1a4e17aa4785a4a8f15156, State: Initialized, Role: FOLLOWER
I20260620 11:42:52.393373 13211 consensus_queue.cc:260] T 00000000000000000000000000000000 P cc8526b37e1a4e17aa4785a4a8f15156 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "cc8526b37e1a4e17aa4785a4a8f15156" member_type: VOTER last_known_addr { host: "127.6.234.254" port: 39029 } }
I20260620 11:42:52.393462 13211 raft_consensus.cc:388] T 00000000000000000000000000000000 P cc8526b37e1a4e17aa4785a4a8f15156 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260620 11:42:52.393507 13211 raft_consensus.cc:482] T 00000000000000000000000000000000 P cc8526b37e1a4e17aa4785a4a8f15156 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260620 11:42:52.393559 13211 raft_consensus.cc:3037] T 00000000000000000000000000000000 P cc8526b37e1a4e17aa4785a4a8f15156 [term 0 FOLLOWER]: Advancing to term 1
I20260620 11:42:52.394222 13211 raft_consensus.cc:504] T 00000000000000000000000000000000 P cc8526b37e1a4e17aa4785a4a8f15156 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "cc8526b37e1a4e17aa4785a4a8f15156" member_type: VOTER last_known_addr { host: "127.6.234.254" port: 39029 } }
I20260620 11:42:52.394344 13211 leader_election.cc:302] T 00000000000000000000000000000000 P cc8526b37e1a4e17aa4785a4a8f15156 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: cc8526b37e1a4e17aa4785a4a8f15156; no voters:
I20260620 11:42:52.394543 13211 leader_election.cc:288] T 00000000000000000000000000000000 P cc8526b37e1a4e17aa4785a4a8f15156 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260620 11:42:52.394590 13216 raft_consensus.cc:2781] T 00000000000000000000000000000000 P cc8526b37e1a4e17aa4785a4a8f15156 [term 1 FOLLOWER]: Leader election won for term 1
I20260620 11:42:52.394706 13216 raft_consensus.cc:686] T 00000000000000000000000000000000 P cc8526b37e1a4e17aa4785a4a8f15156 [term 1 LEADER]: Becoming Leader. State: Replica: cc8526b37e1a4e17aa4785a4a8f15156, State: Running, Role: LEADER
I20260620 11:42:52.394809 13211 sys_catalog.cc:564] T 00000000000000000000000000000000 P cc8526b37e1a4e17aa4785a4a8f15156 [sys.catalog]: configured and running, proceeding with master startup.
I20260620 11:42:52.394905 13216 consensus_queue.cc:237] T 00000000000000000000000000000000 P cc8526b37e1a4e17aa4785a4a8f15156 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "cc8526b37e1a4e17aa4785a4a8f15156" member_type: VOTER last_known_addr { host: "127.6.234.254" port: 39029 } }
I20260620 11:42:52.396035 13217 sys_catalog.cc:455] T 00000000000000000000000000000000 P cc8526b37e1a4e17aa4785a4a8f15156 [sys.catalog]: SysCatalogTable state changed. Reason: New leader cc8526b37e1a4e17aa4785a4a8f15156. Latest consensus state: current_term: 1 leader_uuid: "cc8526b37e1a4e17aa4785a4a8f15156" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "cc8526b37e1a4e17aa4785a4a8f15156" member_type: VOTER last_known_addr { host: "127.6.234.254" port: 39029 } } }
I20260620 11:42:52.396118 13217 sys_catalog.cc:458] T 00000000000000000000000000000000 P cc8526b37e1a4e17aa4785a4a8f15156 [sys.catalog]: This master's current role is: LEADER
I20260620 11:42:52.396318 13218 sys_catalog.cc:455] T 00000000000000000000000000000000 P cc8526b37e1a4e17aa4785a4a8f15156 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "cc8526b37e1a4e17aa4785a4a8f15156" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "cc8526b37e1a4e17aa4785a4a8f15156" member_type: VOTER last_known_addr { host: "127.6.234.254" port: 39029 } } }
I20260620 11:42:52.396370 13218 sys_catalog.cc:458] T 00000000000000000000000000000000 P cc8526b37e1a4e17aa4785a4a8f15156 [sys.catalog]: This master's current role is: LEADER
I20260620 11:42:52.396508 13232 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260620 11:42:52.396966 13232 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260620 11:42:52.398571 13232 catalog_manager.cc:1347] Generated new cluster ID: 62ddcf685f6646508efb7ea8accaf0cf
I20260620 11:42:52.398625 13232 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
Jun 20 11:42:52 dist-test-slave-cjc6 krb5kdc[13130](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781955772, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.6.234.254@KRBTEST.COM
I20260620 11:42:52.403488 13215 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 11:42:52.391461 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45866 (local address 127.6.234.254:39029)
0620 11:42:52.391650 (+ 189us) server_negotiation.cc:206] Beginning negotiation
0620 11:42:52.391659 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0620 11:42:52.391692 (+ 33us) server_negotiation.cc:407] Connection header received
0620 11:42:52.392935 (+ 1243us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 11:42:52.392942 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 11:42:52.393015 (+ 73us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 11:42:52.393131 (+ 116us) server_negotiation.cc:226] Negotiated authn=SASL
0620 11:42:52.393951 (+ 820us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 11:42:52.395181 (+ 1230us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 11:42:52.396238 (+ 1057us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 11:42:52.396858 (+ 620us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 11:42:52.396970 (+ 112us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 11:42:52.399946 (+ 2976us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 11:42:52.399950 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 11:42:52.399950 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 11:42:52.399961 (+ 11us) server_negotiation.cc:922] calling sasl_server_start()
0620 11:42:52.401869 (+ 1908us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 11:42:52.402289 (+ 420us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 11:42:52.402292 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 11:42:52.402293 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 11:42:52.402393 (+ 100us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 11:42:52.402623 (+ 230us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 11:42:52.402624 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 11:42:52.402624 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 11:42:52.402910 (+ 286us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 11:42:52.403071 (+ 161us) server_negotiation.cc:1035] Waiting for connection context
0620 11:42:52.403191 (+ 120us) server_negotiation.cc:299] Negotiation successful
0620 11:42:52.403312 (+ 121us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":133,"thread_start_us":105,"threads_started":1}
I20260620 11:42:52.418318 13232 catalog_manager.cc:1370] Generated new certificate authority record
I20260620 11:42:52.418915 13232 catalog_manager.cc:1504] Loading token signing keys...
I20260620 11:42:52.425081 13232 catalog_manager.cc:5868] T 00000000000000000000000000000000 P cc8526b37e1a4e17aa4785a4a8f15156: Generated new TSK 0
I20260620 11:42:52.425258 13232 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for oryx/127.6.234.193@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.6.234.193@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.6.234.193 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task1JRg7N/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781955746769051-7083-0/krb5kdc/oryx_127.6.234.193.keytab.
Entry for principal oryx/127.6.234.193 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task1JRg7N/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781955746769051-7083-0/krb5kdc/oryx_127.6.234.193.keytab.
I20260620 11:42:52.452435 7083 external_mini_cluster.cc:1326] Running /tmp/dist-test-task1JRg7N/build/release/bin/kudu
/tmp/dist-test-task1JRg7N/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task1JRg7N/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781955746769051-7083-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-task1JRg7N/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781955746769051-7083-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task1JRg7N/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781955746769051-7083-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-task1JRg7N/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781955746769051-7083-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.6.234.193:0
--local_ip_for_outbound_sockets=127.6.234.193
--webserver_interface=127.6.234.193
--webserver_port=0
--tserver_master_addrs=127.6.234.254:39029
--keytab_file=/tmp/dist-test-task1JRg7N/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781955746769051-7083-0/krb5kdc/oryx_127.6.234.193.keytab
--principal=oryx/127.6.234.193
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.6.234.212:38681
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task1JRg7N/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781955746769051-7083-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task1JRg7N/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781955746769051-7083-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task1JRg7N/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781955746769051-7083-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 11:42:52.531594 13237 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 11:42:52.531838 13237 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 11:42:52.531876 13237 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 11:42:52.533021 13237 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 11:42:52.533090 13237 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 11:42:52.533135 13237 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.6.234.193
I20260620 11:42:52.534614 13237 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.6.234.212:38681
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task1JRg7N/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781955746769051-7083-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-task1JRg7N/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781955746769051-7083-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task1JRg7N/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781955746769051-7083-0/krb5kdc/oryx_127.6.234.193.keytab
--principal=oryx/127.6.234.193
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.6.234.193:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task1JRg7N/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781955746769051-7083-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.6.234.193
--webserver_port=0
--tserver_master_addrs=127.6.234.254:39029
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.13237
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.6.234.193
--log_dir=/tmp/dist-test-task1JRg7N/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781955746769051-7083-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 20 Jun 2026 10:32:37 UTC on ccycloud.quasar-bgyofv.root.comops.site
build id 4457173
I20260620 11:42:52.534868 13237 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 11:42:52.535095 13237 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 11:42:52.540021 13242 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 11:42:52.540009 13245 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 11:42:52.539999 13243 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 11:42:52.540367 13237 server_base.cc:1034] running on GCE node
I20260620 11:42:52.540555 13237 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 11:42:52.540731 13237 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 11:42:52.541872 13237 hybrid_clock.cc:648] HybridClock initialized: now 1781955772541855 us; error 30 us; skew 500 ppm
Jun 20 11:42:52 dist-test-slave-cjc6 krb5kdc[13130](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781955772, etypes {rep=17 tkt=17 ses=17}, oryx/127.6.234.193@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 11:42:52.544391 13237 init.cc:377] Logged in from keytab as oryx/127.6.234.193@KRBTEST.COM (short username oryx)
I20260620 11:42:52.544831 13237 webserver.cc:460] Webserver started at http://127.6.234.193:37865/ using document root <none> and password file <none>
I20260620 11:42:52.545053 13237 fs_manager.cc:359] Metadata directory not provided
I20260620 11:42:52.545120 13237 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 11:42:52.545270 13237 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 11:42:52.546332 13237 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task1JRg7N/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781955746769051-7083-0/minicluster-data/ts-0/data/instance:
uuid: "35c3aac7e3ce4bb3bb0294aa36fd45ab"
format_stamp: "Formatted at 2026-06-20 11:42:52 on dist-test-slave-cjc6"
server_key: "ad11f460cee7d38a2e66cf539338d192"
server_key_iv: "8ea088124828eb1f37680019cf3038bd"
server_key_version: "encryptionkey@0"
I20260620 11:42:52.546754 13237 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task1JRg7N/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781955746769051-7083-0/minicluster-data/ts-0/wal/instance:
uuid: "35c3aac7e3ce4bb3bb0294aa36fd45ab"
format_stamp: "Formatted at 2026-06-20 11:42:52 on dist-test-slave-cjc6"
server_key: "ad11f460cee7d38a2e66cf539338d192"
server_key_iv: "8ea088124828eb1f37680019cf3038bd"
server_key_version: "encryptionkey@0"
I20260620 11:42:52.548213 13237 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260620 11:42:52.548928 13252 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 11:42:52.549093 13237 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260620 11:42:52.549186 13237 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task1JRg7N/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781955746769051-7083-0/minicluster-data/ts-0/data,/tmp/dist-test-task1JRg7N/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781955746769051-7083-0/minicluster-data/ts-0/wal
uuid: "35c3aac7e3ce4bb3bb0294aa36fd45ab"
format_stamp: "Formatted at 2026-06-20 11:42:52 on dist-test-slave-cjc6"
server_key: "ad11f460cee7d38a2e66cf539338d192"
server_key_iv: "8ea088124828eb1f37680019cf3038bd"
server_key_version: "encryptionkey@0"
I20260620 11:42:52.549268 13237 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task1JRg7N/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781955746769051-7083-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-task1JRg7N/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781955746769051-7083-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-task1JRg7N/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781955746769051-7083-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 11:42:52.558139 13237 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 11:42:52.559870 13237 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 11:42:52.560029 13237 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 11:42:52.560243 13237 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 11:42:52.560539 13237 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 11:42:52.560595 13237 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 11:42:52.560634 13237 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 11:42:52.560662 13237 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 11:42:52.566461 13237 rpc_server.cc:307] RPC server started. Bound to: 127.6.234.193:38171
I20260620 11:42:52.566509 13365 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.6.234.193:38171 every 8 connection(s)
I20260620 11:42:52.566799 13237 server_base.cc:1166] Dumped server information to /tmp/dist-test-task1JRg7N/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781955746769051-7083-0/minicluster-data/ts-0/data/info.pb
I20260620 11:42:52.567137 7083 external_mini_cluster.cc:1388] Started /tmp/dist-test-task1JRg7N/build/release/bin/kudu as pid 13237
I20260620 11:42:52.567250 7083 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task1JRg7N/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781955746769051-7083-0/minicluster-data/ts-0/wal/instance
I20260620 11:42:52.567467 7083 external_mini_cluster.cc:1428] Setting key 873bde4ae4cdf9a0044ce579b912fbb8
Jun 20 11:42:52 dist-test-slave-cjc6 krb5kdc[13130](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781955772, etypes {rep=17 tkt=17 ses=17}, oryx/127.6.234.193@KRBTEST.COM for oryx/127.6.234.254@KRBTEST.COM
WARNING: no policy specified for oryx/127.6.234.194@KRBTEST.COM; defaulting to no policy
I20260620 11:42:52.576561 13215 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 11:42:52.567456 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.6.234.193:39403 (local address 127.6.234.254:39029)
0620 11:42:52.567498 (+ 42us) server_negotiation.cc:206] Beginning negotiation
0620 11:42:52.567501 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0620 11:42:52.567747 (+ 246us) server_negotiation.cc:407] Connection header received
0620 11:42:52.568349 (+ 602us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 11:42:52.568351 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 11:42:52.568363 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 11:42:52.568424 (+ 61us) server_negotiation.cc:226] Negotiated authn=SASL
0620 11:42:52.569448 (+ 1024us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 11:42:52.570024 (+ 576us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 11:42:52.571081 (+ 1057us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 11:42:52.571436 (+ 355us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 11:42:52.571514 (+ 78us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 11:42:52.573887 (+ 2373us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 11:42:52.573889 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 11:42:52.573889 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 11:42:52.573895 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0620 11:42:52.575373 (+ 1478us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 11:42:52.575756 (+ 383us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 11:42:52.575756 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 11:42:52.575757 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 11:42:52.575805 (+ 48us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 11:42:52.576008 (+ 203us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 11:42:52.576008 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 11:42:52.576009 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 11:42:52.576102 (+ 93us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 11:42:52.576177 (+ 75us) server_negotiation.cc:1035] Waiting for connection context
0620 11:42:52.576430 (+ 253us) server_negotiation.cc:299] Negotiation successful
0620 11:42:52.576498 (+ 68us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":19}
I20260620 11:42:52.576684 13368 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 11:42:52.567451 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.234.254:39029 (local address 127.6.234.193:39403)
0620 11:42:52.567604 (+ 153us) negotiation.cc:107] Waiting for socket to connect
0620 11:42:52.567668 (+ 64us) client_negotiation.cc:174] Beginning negotiation
0620 11:42:52.568255 (+ 587us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 11:42:52.568490 (+ 235us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 11:42:52.568494 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 11:42:52.569060 (+ 566us) client_negotiation.cc:189] Negotiated authn=SASL
0620 11:42:52.569348 (+ 288us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 11:42:52.569354 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 11:42:52.570114 (+ 760us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 11:42:52.570115 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 11:42:52.570993 (+ 878us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 11:42:52.570996 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 11:42:52.571519 (+ 523us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 11:42:52.571520 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 11:42:52.571616 (+ 96us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 11:42:52.572135 (+ 519us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 11:42:52.572149 (+ 14us) client_negotiation.cc:656] calling sasl_client_start()
0620 11:42:52.573803 (+ 1654us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 11:42:52.575453 (+ 1650us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 11:42:52.575460 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 11:42:52.575463 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0620 11:42:52.575694 (+ 231us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 11:42:52.575864 (+ 170us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 11:42:52.575864 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 11:42:52.575865 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0620 11:42:52.575957 (+ 92us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 11:42:52.576158 (+ 201us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 11:42:52.576162 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 11:42:52.576309 (+ 147us) client_negotiation.cc:769] Sending connection context
0620 11:42:52.576398 (+ 89us) client_negotiation.cc:240] Negotiation successful
0620 11:42:52.576507 (+ 109us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":113,"thread_start_us":86,"threads_started":1}
I20260620 11:42:52.577193 13366 heartbeater.cc:347] Connected to a master server at 127.6.234.254:39029
I20260620 11:42:52.577303 13366 heartbeater.cc:464] Registering TS with master...
I20260620 11:42:52.577497 13366 heartbeater.cc:511] Master 127.6.234.254:39029 requested a full tablet report, sending...
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.6.234.194@KRBTEST.COM" created.
I20260620 11:42:52.578116 13175 ts_manager.cc:194] Registered new tserver with Master: 35c3aac7e3ce4bb3bb0294aa36fd45ab (127.6.234.193:38171)
I20260620 11:42:52.578719 13175 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.6.234.193@KRBTEST.COM'} at 127.6.234.193:39403
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.6.234.194 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task1JRg7N/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781955746769051-7083-0/krb5kdc/oryx_127.6.234.194.keytab.
Entry for principal oryx/127.6.234.194 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task1JRg7N/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781955746769051-7083-0/krb5kdc/oryx_127.6.234.194.keytab.
I20260620 11:42:52.590726 7083 external_mini_cluster.cc:1326] Running /tmp/dist-test-task1JRg7N/build/release/bin/kudu
/tmp/dist-test-task1JRg7N/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task1JRg7N/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781955746769051-7083-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-task1JRg7N/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781955746769051-7083-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task1JRg7N/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781955746769051-7083-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-task1JRg7N/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781955746769051-7083-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.6.234.194:0
--local_ip_for_outbound_sockets=127.6.234.194
--webserver_interface=127.6.234.194
--webserver_port=0
--tserver_master_addrs=127.6.234.254:39029
--keytab_file=/tmp/dist-test-task1JRg7N/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781955746769051-7083-0/krb5kdc/oryx_127.6.234.194.keytab
--principal=oryx/127.6.234.194
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.6.234.212:38681
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task1JRg7N/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781955746769051-7083-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task1JRg7N/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781955746769051-7083-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task1JRg7N/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781955746769051-7083-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 11:42:52.670141 13371 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 11:42:52.670320 13371 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 11:42:52.670341 13371 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 11:42:52.671414 13371 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 11:42:52.671454 13371 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 11:42:52.671506 13371 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.6.234.194
I20260620 11:42:52.672859 13371 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.6.234.212:38681
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task1JRg7N/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781955746769051-7083-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-task1JRg7N/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781955746769051-7083-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task1JRg7N/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781955746769051-7083-0/krb5kdc/oryx_127.6.234.194.keytab
--principal=oryx/127.6.234.194
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.6.234.194:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task1JRg7N/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781955746769051-7083-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.6.234.194
--webserver_port=0
--tserver_master_addrs=127.6.234.254:39029
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.13371
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.6.234.194
--log_dir=/tmp/dist-test-task1JRg7N/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781955746769051-7083-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 20 Jun 2026 10:32:37 UTC on ccycloud.quasar-bgyofv.root.comops.site
build id 4457173
I20260620 11:42:52.673090 13371 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 11:42:52.673306 13371 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 11:42:52.678058 13379 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 11:42:52.678082 13377 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 11:42:52.678113 13376 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 11:42:52.678272 13371 server_base.cc:1034] running on GCE node
I20260620 11:42:52.678478 13371 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 11:42:52.678659 13371 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 11:42:52.679801 13371 hybrid_clock.cc:648] HybridClock initialized: now 1781955772679792 us; error 26 us; skew 500 ppm
Jun 20 11:42:52 dist-test-slave-cjc6 krb5kdc[13130](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781955772, etypes {rep=17 tkt=17 ses=17}, oryx/127.6.234.194@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 11:42:52.682189 13371 init.cc:377] Logged in from keytab as oryx/127.6.234.194@KRBTEST.COM (short username oryx)
I20260620 11:42:52.682596 13371 webserver.cc:460] Webserver started at http://127.6.234.194:41999/ using document root <none> and password file <none>
I20260620 11:42:52.682806 13371 fs_manager.cc:359] Metadata directory not provided
I20260620 11:42:52.682850 13371 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 11:42:52.682992 13371 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 11:42:52.683964 13371 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task1JRg7N/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781955746769051-7083-0/minicluster-data/ts-1/data/instance:
uuid: "553457094e274e55ace2e66d211e3171"
format_stamp: "Formatted at 2026-06-20 11:42:52 on dist-test-slave-cjc6"
server_key: "f8874bb3589fc5275bc848d8f9df445f"
server_key_iv: "75a5b4f95369ca6d8a94c9ea8b643d10"
server_key_version: "encryptionkey@0"
I20260620 11:42:52.684302 13371 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task1JRg7N/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781955746769051-7083-0/minicluster-data/ts-1/wal/instance:
uuid: "553457094e274e55ace2e66d211e3171"
format_stamp: "Formatted at 2026-06-20 11:42:52 on dist-test-slave-cjc6"
server_key: "f8874bb3589fc5275bc848d8f9df445f"
server_key_iv: "75a5b4f95369ca6d8a94c9ea8b643d10"
server_key_version: "encryptionkey@0"
I20260620 11:42:52.685554 13371 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.001s
I20260620 11:42:52.686314 13386 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 11:42:52.686512 13371 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.001s
I20260620 11:42:52.686599 13371 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task1JRg7N/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781955746769051-7083-0/minicluster-data/ts-1/data,/tmp/dist-test-task1JRg7N/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781955746769051-7083-0/minicluster-data/ts-1/wal
uuid: "553457094e274e55ace2e66d211e3171"
format_stamp: "Formatted at 2026-06-20 11:42:52 on dist-test-slave-cjc6"
server_key: "f8874bb3589fc5275bc848d8f9df445f"
server_key_iv: "75a5b4f95369ca6d8a94c9ea8b643d10"
server_key_version: "encryptionkey@0"
I20260620 11:42:52.686668 13371 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task1JRg7N/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781955746769051-7083-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-task1JRg7N/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781955746769051-7083-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-task1JRg7N/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781955746769051-7083-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 11:42:52.696666 13371 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 11:42:52.699007 13371 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 11:42:52.699157 13371 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 11:42:52.699378 13371 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 11:42:52.699646 13371 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 11:42:52.699678 13371 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 11:42:52.699700 13371 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 11:42:52.699716 13371 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 11:42:52.705470 13371 rpc_server.cc:307] RPC server started. Bound to: 127.6.234.194:39263
I20260620 11:42:52.705513 13499 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.6.234.194:39263 every 8 connection(s)
I20260620 11:42:52.705876 13371 server_base.cc:1166] Dumped server information to /tmp/dist-test-task1JRg7N/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781955746769051-7083-0/minicluster-data/ts-1/data/info.pb
Jun 20 11:42:52 dist-test-slave-cjc6 krb5kdc[13130](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781955772, etypes {rep=17 tkt=17 ses=17}, oryx/127.6.234.194@KRBTEST.COM for oryx/127.6.234.254@KRBTEST.COM
I20260620 11:42:52.714474 13215 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 11:42:52.706582 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.6.234.194:55697 (local address 127.6.234.254:39029)
0620 11:42:52.706639 (+ 57us) server_negotiation.cc:206] Beginning negotiation
0620 11:42:52.706642 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0620 11:42:52.706851 (+ 209us) server_negotiation.cc:407] Connection header received
0620 11:42:52.707314 (+ 463us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 11:42:52.707316 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 11:42:52.707325 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 11:42:52.707382 (+ 57us) server_negotiation.cc:226] Negotiated authn=SASL
0620 11:42:52.708262 (+ 880us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 11:42:52.708747 (+ 485us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 11:42:52.709735 (+ 988us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 11:42:52.709949 (+ 214us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 11:42:52.710002 (+ 53us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 11:42:52.712084 (+ 2082us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 11:42:52.712085 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 11:42:52.712085 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 11:42:52.712091 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0620 11:42:52.713238 (+ 1147us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 11:42:52.713601 (+ 363us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 11:42:52.713602 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 11:42:52.713602 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 11:42:52.713648 (+ 46us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 11:42:52.713928 (+ 280us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 11:42:52.713929 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 11:42:52.713930 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 11:42:52.714053 (+ 123us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 11:42:52.714118 (+ 65us) server_negotiation.cc:1035] Waiting for connection context
0620 11:42:52.714348 (+ 230us) server_negotiation.cc:299] Negotiation successful
0620 11:42:52.714411 (+ 63us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":32}
I20260620 11:42:52.714669 13502 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 11:42:52.706533 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.234.254:39029 (local address 127.6.234.194:55697)
0620 11:42:52.706693 (+ 160us) negotiation.cc:107] Waiting for socket to connect
0620 11:42:52.706734 (+ 41us) client_negotiation.cc:174] Beginning negotiation
0620 11:42:52.707232 (+ 498us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 11:42:52.707434 (+ 202us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 11:42:52.707437 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 11:42:52.708005 (+ 568us) client_negotiation.cc:189] Negotiated authn=SASL
0620 11:42:52.708188 (+ 183us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 11:42:52.708192 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 11:42:52.708837 (+ 645us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 11:42:52.708838 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 11:42:52.709620 (+ 782us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 11:42:52.709622 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 11:42:52.710020 (+ 398us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 11:42:52.710020 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 11:42:52.710121 (+ 101us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 11:42:52.710579 (+ 458us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 11:42:52.710587 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0620 11:42:52.711995 (+ 1408us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 11:42:52.713309 (+ 1314us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 11:42:52.713314 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 11:42:52.713316 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0620 11:42:52.713538 (+ 222us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 11:42:52.713748 (+ 210us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 11:42:52.713748 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 11:42:52.713749 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0620 11:42:52.713846 (+ 97us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 11:42:52.714120 (+ 274us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 11:42:52.714124 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 11:42:52.714233 (+ 109us) client_negotiation.cc:769] Sending connection context
0620 11:42:52.714315 (+ 82us) client_negotiation.cc:240] Negotiation successful
0620 11:42:52.714437 (+ 122us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":121,"thread_start_us":90,"threads_started":1}
I20260620 11:42:52.715071 7083 external_mini_cluster.cc:1388] Started /tmp/dist-test-task1JRg7N/build/release/bin/kudu as pid 13371
I20260620 11:42:52.715186 7083 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task1JRg7N/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781955746769051-7083-0/minicluster-data/ts-1/wal/instance
I20260620 11:42:52.715317 13500 heartbeater.cc:347] Connected to a master server at 127.6.234.254:39029
I20260620 11:42:52.715368 7083 external_mini_cluster.cc:1428] Setting key d2ad619972b5ef0d71e262f2d3f56e75
I20260620 11:42:52.715426 13500 heartbeater.cc:464] Registering TS with master...
I20260620 11:42:52.715611 13500 heartbeater.cc:511] Master 127.6.234.254:39029 requested a full tablet report, sending...
I20260620 11:42:52.715956 13175 ts_manager.cc:194] Registered new tserver with Master: 553457094e274e55ace2e66d211e3171 (127.6.234.194:39263)
I20260620 11:42:52.716336 13175 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.6.234.194@KRBTEST.COM'} at 127.6.234.194:55697
WARNING: no policy specified for oryx/127.6.234.195@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.6.234.195@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.6.234.195 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task1JRg7N/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781955746769051-7083-0/krb5kdc/oryx_127.6.234.195.keytab.
Entry for principal oryx/127.6.234.195 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task1JRg7N/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781955746769051-7083-0/krb5kdc/oryx_127.6.234.195.keytab.
I20260620 11:42:52.739849 7083 external_mini_cluster.cc:1326] Running /tmp/dist-test-task1JRg7N/build/release/bin/kudu
/tmp/dist-test-task1JRg7N/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task1JRg7N/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781955746769051-7083-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-task1JRg7N/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781955746769051-7083-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task1JRg7N/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781955746769051-7083-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-task1JRg7N/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781955746769051-7083-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.6.234.195:0
--local_ip_for_outbound_sockets=127.6.234.195
--webserver_interface=127.6.234.195
--webserver_port=0
--tserver_master_addrs=127.6.234.254:39029
--keytab_file=/tmp/dist-test-task1JRg7N/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781955746769051-7083-0/krb5kdc/oryx_127.6.234.195.keytab
--principal=oryx/127.6.234.195
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.6.234.212:38681
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task1JRg7N/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781955746769051-7083-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task1JRg7N/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781955746769051-7083-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task1JRg7N/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781955746769051-7083-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 11:42:52.821022 13505 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 11:42:52.821213 13505 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 11:42:52.821235 13505 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 11:42:52.822383 13505 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 11:42:52.822425 13505 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 11:42:52.822451 13505 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.6.234.195
I20260620 11:42:52.823799 13505 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.6.234.212:38681
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task1JRg7N/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781955746769051-7083-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-task1JRg7N/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781955746769051-7083-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task1JRg7N/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781955746769051-7083-0/krb5kdc/oryx_127.6.234.195.keytab
--principal=oryx/127.6.234.195
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.6.234.195:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task1JRg7N/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781955746769051-7083-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.6.234.195
--webserver_port=0
--tserver_master_addrs=127.6.234.254:39029
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.13505
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.6.234.195
--log_dir=/tmp/dist-test-task1JRg7N/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781955746769051-7083-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 20 Jun 2026 10:32:37 UTC on ccycloud.quasar-bgyofv.root.comops.site
build id 4457173
I20260620 11:42:52.823971 13505 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 11:42:52.824132 13505 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 11:42:52.828940 13513 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 11:42:52.829027 13511 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 11:42:52.829025 13510 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 11:42:52.829191 13505 server_base.cc:1034] running on GCE node
I20260620 11:42:52.829384 13505 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 11:42:52.829584 13505 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 11:42:52.830721 13505 hybrid_clock.cc:648] HybridClock initialized: now 1781955772830706 us; error 29 us; skew 500 ppm
Jun 20 11:42:52 dist-test-slave-cjc6 krb5kdc[13130](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781955772, etypes {rep=17 tkt=17 ses=17}, oryx/127.6.234.195@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 11:42:52.833144 13505 init.cc:377] Logged in from keytab as oryx/127.6.234.195@KRBTEST.COM (short username oryx)
I20260620 11:42:52.833609 13505 webserver.cc:460] Webserver started at http://127.6.234.195:35099/ using document root <none> and password file <none>
I20260620 11:42:52.833861 13505 fs_manager.cc:359] Metadata directory not provided
I20260620 11:42:52.833935 13505 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 11:42:52.834025 13505 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 11:42:52.834987 13505 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task1JRg7N/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781955746769051-7083-0/minicluster-data/ts-2/data/instance:
uuid: "212956ba6c57456f9d631b576e69d6a2"
format_stamp: "Formatted at 2026-06-20 11:42:52 on dist-test-slave-cjc6"
server_key: "4b372029a0757ec2ccfe20e0cd19ba6c"
server_key_iv: "8f99d304325e013005332a7f4be9f355"
server_key_version: "encryptionkey@0"
I20260620 11:42:52.835273 13505 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task1JRg7N/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781955746769051-7083-0/minicluster-data/ts-2/wal/instance:
uuid: "212956ba6c57456f9d631b576e69d6a2"
format_stamp: "Formatted at 2026-06-20 11:42:52 on dist-test-slave-cjc6"
server_key: "4b372029a0757ec2ccfe20e0cd19ba6c"
server_key_iv: "8f99d304325e013005332a7f4be9f355"
server_key_version: "encryptionkey@0"
I20260620 11:42:52.836503 13505 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260620 11:42:52.837219 13520 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 11:42:52.837471 13505 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260620 11:42:52.837574 13505 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task1JRg7N/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781955746769051-7083-0/minicluster-data/ts-2/data,/tmp/dist-test-task1JRg7N/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781955746769051-7083-0/minicluster-data/ts-2/wal
uuid: "212956ba6c57456f9d631b576e69d6a2"
format_stamp: "Formatted at 2026-06-20 11:42:52 on dist-test-slave-cjc6"
server_key: "4b372029a0757ec2ccfe20e0cd19ba6c"
server_key_iv: "8f99d304325e013005332a7f4be9f355"
server_key_version: "encryptionkey@0"
I20260620 11:42:52.837646 13505 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task1JRg7N/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781955746769051-7083-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-task1JRg7N/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781955746769051-7083-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-task1JRg7N/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781955746769051-7083-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 11:42:52.850934 13505 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 11:42:52.852922 13505 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 11:42:52.853087 13505 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 11:42:52.853343 13505 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 11:42:52.853680 13505 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 11:42:52.853823 13505 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 11:42:52.853868 13505 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 11:42:52.853894 13505 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 11:42:52.859615 13505 rpc_server.cc:307] RPC server started. Bound to: 127.6.234.195:39097
I20260620 11:42:52.859659 13633 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.6.234.195:39097 every 8 connection(s)
I20260620 11:42:52.859990 13505 server_base.cc:1166] Dumped server information to /tmp/dist-test-task1JRg7N/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781955746769051-7083-0/minicluster-data/ts-2/data/info.pb
I20260620 11:42:52.864262 7083 external_mini_cluster.cc:1388] Started /tmp/dist-test-task1JRg7N/build/release/bin/kudu as pid 13505
I20260620 11:42:52.864373 7083 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task1JRg7N/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781955746769051-7083-0/minicluster-data/ts-2/wal/instance
I20260620 11:42:52.864567 7083 external_mini_cluster.cc:1428] Setting key 611d0a038a5f54e8e6d40acae7339046
Jun 20 11:42:52 dist-test-slave-cjc6 krb5kdc[13130](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781955772, etypes {rep=17 tkt=17 ses=17}, oryx/127.6.234.195@KRBTEST.COM for oryx/127.6.234.254@KRBTEST.COM
I20260620 11:42:52.869249 13215 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 11:42:52.860713 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.6.234.195:57451 (local address 127.6.234.254:39029)
0620 11:42:52.860783 (+ 70us) server_negotiation.cc:206] Beginning negotiation
0620 11:42:52.860786 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0620 11:42:52.860987 (+ 201us) server_negotiation.cc:407] Connection header received
0620 11:42:52.861451 (+ 464us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 11:42:52.861452 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 11:42:52.861461 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 11:42:52.861518 (+ 57us) server_negotiation.cc:226] Negotiated authn=SASL
0620 11:42:52.862576 (+ 1058us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 11:42:52.863124 (+ 548us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 11:42:52.864056 (+ 932us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 11:42:52.864398 (+ 342us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 11:42:52.864475 (+ 77us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 11:42:52.866873 (+ 2398us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 11:42:52.866875 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 11:42:52.866875 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 11:42:52.866881 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0620 11:42:52.868057 (+ 1176us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 11:42:52.868509 (+ 452us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 11:42:52.868510 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 11:42:52.868510 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 11:42:52.868560 (+ 50us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 11:42:52.868763 (+ 203us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 11:42:52.868763 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 11:42:52.868763 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 11:42:52.868859 (+ 96us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 11:42:52.868907 (+ 48us) server_negotiation.cc:1035] Waiting for connection context
0620 11:42:52.869131 (+ 224us) server_negotiation.cc:299] Negotiation successful
0620 11:42:52.869188 (+ 57us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":42}
I20260620 11:42:52.869467 13636 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 11:42:52.860692 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.234.254:39029 (local address 127.6.234.195:57451)
0620 11:42:52.860853 (+ 161us) negotiation.cc:107] Waiting for socket to connect
0620 11:42:52.860895 (+ 42us) client_negotiation.cc:174] Beginning negotiation
0620 11:42:52.861369 (+ 474us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 11:42:52.861571 (+ 202us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 11:42:52.861574 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 11:42:52.862242 (+ 668us) client_negotiation.cc:189] Negotiated authn=SASL
0620 11:42:52.862484 (+ 242us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 11:42:52.862490 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 11:42:52.863206 (+ 716us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 11:42:52.863207 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 11:42:52.863981 (+ 774us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 11:42:52.863983 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 11:42:52.864493 (+ 510us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 11:42:52.864494 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 11:42:52.864738 (+ 244us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 11:42:52.865319 (+ 581us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 11:42:52.865327 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0620 11:42:52.866793 (+ 1466us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 11:42:52.868215 (+ 1422us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 11:42:52.868221 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 11:42:52.868223 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0620 11:42:52.868442 (+ 219us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 11:42:52.868618 (+ 176us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 11:42:52.868619 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 11:42:52.868619 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0620 11:42:52.868710 (+ 91us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 11:42:52.868912 (+ 202us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 11:42:52.868916 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 11:42:52.869025 (+ 109us) client_negotiation.cc:769] Sending connection context
0620 11:42:52.869106 (+ 81us) client_negotiation.cc:240] Negotiation successful
0620 11:42:52.869219 (+ 113us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":116,"thread_start_us":90,"threads_started":1}
I20260620 11:42:52.869979 13634 heartbeater.cc:347] Connected to a master server at 127.6.234.254:39029
I20260620 11:42:52.870074 13634 heartbeater.cc:464] Registering TS with master...
I20260620 11:42:52.870209 13634 heartbeater.cc:511] Master 127.6.234.254:39029 requested a full tablet report, sending...
I20260620 11:42:52.870570 13175 ts_manager.cc:194] Registered new tserver with Master: 212956ba6c57456f9d631b576e69d6a2 (127.6.234.195:39097)
I20260620 11:42:52.870966 13175 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.6.234.195@KRBTEST.COM'} at 127.6.234.195:57451
I20260620 11:42:52.876260 7083 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
WARNING: no policy specified for oryx/127.6.234.253@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.6.234.253@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.6.234.253 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task1JRg7N/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781955746769051-7083-0/krb5kdc/oryx_127.6.234.253.keytab.
Entry for principal oryx/127.6.234.253 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task1JRg7N/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781955746769051-7083-0/krb5kdc/oryx_127.6.234.253.keytab.
I20260620 11:42:52.900915 7083 external_mini_cluster.cc:1326] Running /tmp/dist-test-task1JRg7N/build/release/bin/kudu
/tmp/dist-test-task1JRg7N/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task1JRg7N/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781955746769051-7083-0/minicluster-data/master-1/wal
--fs_data_dirs=/tmp/dist-test-task1JRg7N/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781955746769051-7083-0/minicluster-data/master-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task1JRg7N/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781955746769051-7083-0/minicluster-data/master-1/logs
--server_dump_info_path=/tmp/dist-test-task1JRg7N/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781955746769051-7083-0/minicluster-data/master-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.6.234.253:41509
--webserver_interface=127.6.234.253
--webserver_port=0
--keytab_file=/tmp/dist-test-task1JRg7N/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781955746769051-7083-0/krb5kdc/oryx_127.6.234.253.keytab
--principal=oryx/127.6.234.253
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.6.234.212:38681
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.6.234.254:39029,127.6.234.253:41509
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task1JRg7N/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781955746769051-7083-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task1JRg7N/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781955746769051-7083-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task1JRg7N/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781955746769051-7083-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 11:42:52.981571 13639 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 11:42:52.981791 13639 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 11:42:52.981813 13639 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 11:42:52.982978 13639 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 11:42:52.983021 13639 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 11:42:52.983035 13639 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 11:42:52.983047 13639 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 11:42:52.983059 13639 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 11:42:52.984704 13639 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.6.234.212:38681
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task1JRg7N/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781955746769051-7083-0/minicluster-data/master-1/data
--fs_wal_dir=/tmp/dist-test-task1JRg7N/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781955746769051-7083-0/minicluster-data/master-1/wal
--ipki_ca_key_size=768
--master_addresses=127.6.234.254:39029,127.6.234.253:41509
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task1JRg7N/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781955746769051-7083-0/krb5kdc/oryx_127.6.234.253.keytab
--principal=oryx/127.6.234.253
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.6.234.253:41509
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task1JRg7N/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781955746769051-7083-0/minicluster-data/master-1/data/info.pb
--webserver_interface=127.6.234.253
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.13639
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task1JRg7N/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781955746769051-7083-0/minicluster-data/master-1/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 20 Jun 2026 10:32:37 UTC on ccycloud.quasar-bgyofv.root.comops.site
build id 4457173
W20260620 11:42:52.984843 13639 master_options.cc:55] Only 2 masters are specified by master_addresses_flag ('127.6.234.254:39029,127.6.234.253:41509'), but minimum of 3 are required to tolerate failures of any one master. It is recommended to use at least 3 masters.
I20260620 11:42:52.985002 13639 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 11:42:52.985260 13639 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 11:42:52.989979 13645 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 11:42:52.990051 13647 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 11:42:52.990026 13644 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 11:42:52.990125 13639 server_base.cc:1034] running on GCE node
I20260620 11:42:52.990422 13639 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 11:42:52.990675 13639 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 11:42:52.991868 13639 hybrid_clock.cc:648] HybridClock initialized: now 1781955772991825 us; error 31 us; skew 500 ppm
Jun 20 11:42:52 dist-test-slave-cjc6 krb5kdc[13130](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781955772, etypes {rep=17 tkt=17 ses=17}, oryx/127.6.234.253@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 11:42:52.994307 13639 init.cc:377] Logged in from keytab as oryx/127.6.234.253@KRBTEST.COM (short username oryx)
I20260620 11:42:52.994936 13639 webserver.cc:460] Webserver started at http://127.6.234.253:44945/ using document root <none> and password file <none>
I20260620 11:42:52.995191 13639 fs_manager.cc:359] Metadata directory not provided
I20260620 11:42:52.995234 13639 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 11:42:52.995338 13639 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 11:42:52.996462 13639 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task1JRg7N/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781955746769051-7083-0/minicluster-data/master-1/data/instance:
uuid: "4312e033271940b593bea0332e185bff"
format_stamp: "Formatted at 2026-06-20 11:42:52 on dist-test-slave-cjc6"
server_key: "e2dc6fee535f92e5e4241117a118701d"
server_key_iv: "d0867c552837f801b2336ce81ba05922"
server_key_version: "encryptionkey@0"
I20260620 11:42:52.996754 13639 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task1JRg7N/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781955746769051-7083-0/minicluster-data/master-1/wal/instance:
uuid: "4312e033271940b593bea0332e185bff"
format_stamp: "Formatted at 2026-06-20 11:42:52 on dist-test-slave-cjc6"
server_key: "e2dc6fee535f92e5e4241117a118701d"
server_key_iv: "d0867c552837f801b2336ce81ba05922"
server_key_version: "encryptionkey@0"
I20260620 11:42:52.998034 13639 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.003s
I20260620 11:42:52.998704 13654 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 11:42:52.998898 13639 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.001s
I20260620 11:42:52.998967 13639 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task1JRg7N/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781955746769051-7083-0/minicluster-data/master-1/data,/tmp/dist-test-task1JRg7N/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781955746769051-7083-0/minicluster-data/master-1/wal
uuid: "4312e033271940b593bea0332e185bff"
format_stamp: "Formatted at 2026-06-20 11:42:52 on dist-test-slave-cjc6"
server_key: "e2dc6fee535f92e5e4241117a118701d"
server_key_iv: "d0867c552837f801b2336ce81ba05922"
server_key_version: "encryptionkey@0"
I20260620 11:42:52.999053 13639 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task1JRg7N/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781955746769051-7083-0/minicluster-data/master-1/wal
metadata directory: /tmp/dist-test-task1JRg7N/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781955746769051-7083-0/minicluster-data/master-1/wal
1 data directories: /tmp/dist-test-task1JRg7N/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781955746769051-7083-0/minicluster-data/master-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 11:42:53.012804 13639 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 11:42:53.014605 13639 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 11:42:53.014750 13639 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 11:42:53.018587 13639 rpc_server.cc:307] RPC server started. Bound to: 127.6.234.253:41509
I20260620 11:42:53.018667 13706 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.6.234.253:41509 every 8 connection(s)
I20260620 11:42:53.018996 13639 server_base.cc:1166] Dumped server information to /tmp/dist-test-task1JRg7N/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781955746769051-7083-0/minicluster-data/master-1/data/info.pb
I20260620 11:42:53.019892 13707 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 11:42:53.021369 13707 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.6.234.254" port: 39029 } has no permanent_uuid. Determining permanent_uuid...
I20260620 11:42:53.025425 7083 external_mini_cluster.cc:1388] Started /tmp/dist-test-task1JRg7N/build/release/bin/kudu as pid 13639
Jun 20 11:42:53 dist-test-slave-cjc6 krb5kdc[13130](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781955772, etypes {rep=17 tkt=17 ses=17}, oryx/127.6.234.253@KRBTEST.COM for oryx/127.6.234.254@KRBTEST.COM
I20260620 11:42:53.030617 13215 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 11:42:53.022080 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51572 (local address 127.6.234.254:39029)
0620 11:42:53.022145 (+ 65us) server_negotiation.cc:206] Beginning negotiation
0620 11:42:53.022147 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 11:42:53.022346 (+ 199us) server_negotiation.cc:407] Connection header received
0620 11:42:53.022781 (+ 435us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 11:42:53.022782 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 11:42:53.022791 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 11:42:53.022846 (+ 55us) server_negotiation.cc:226] Negotiated authn=SASL
0620 11:42:53.023371 (+ 525us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 11:42:53.023849 (+ 478us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 11:42:53.024767 (+ 918us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 11:42:53.024951 (+ 184us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 11:42:53.025000 (+ 49us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 11:42:53.027524 (+ 2524us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 11:42:53.027526 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 11:42:53.027526 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 11:42:53.027532 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0620 11:42:53.029024 (+ 1492us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 11:42:53.029489 (+ 465us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 11:42:53.029490 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 11:42:53.029490 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 11:42:53.029567 (+ 77us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 11:42:53.029886 (+ 319us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 11:42:53.029887 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 11:42:53.029888 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 11:42:53.030108 (+ 220us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 11:42:53.030195 (+ 87us) server_negotiation.cc:1035] Waiting for connection context
0620 11:42:53.030433 (+ 238us) server_negotiation.cc:299] Negotiation successful
0620 11:42:53.030519 (+ 86us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":40}
I20260620 11:42:53.030635 13709 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 11:42:53.022053 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.234.254:39029 (local address 127.0.0.1:51572)
0620 11:42:53.022223 (+ 170us) negotiation.cc:107] Waiting for socket to connect
0620 11:42:53.022262 (+ 39us) client_negotiation.cc:174] Beginning negotiation
0620 11:42:53.022695 (+ 433us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 11:42:53.022895 (+ 200us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 11:42:53.022898 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 11:42:53.023122 (+ 224us) client_negotiation.cc:189] Negotiated authn=SASL
0620 11:42:53.023302 (+ 180us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 11:42:53.023306 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 11:42:53.023926 (+ 620us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 11:42:53.023926 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 11:42:53.024694 (+ 768us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 11:42:53.024696 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 11:42:53.025018 (+ 322us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 11:42:53.025019 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 11:42:53.025118 (+ 99us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 11:42:53.025638 (+ 520us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 11:42:53.025649 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0620 11:42:53.027439 (+ 1790us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 11:42:53.029095 (+ 1656us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 11:42:53.029099 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 11:42:53.029101 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0620 11:42:53.029382 (+ 281us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 11:42:53.029643 (+ 261us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 11:42:53.029644 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 11:42:53.029644 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0620 11:42:53.029816 (+ 172us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 11:42:53.030186 (+ 370us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 11:42:53.030190 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 11:42:53.030301 (+ 111us) client_negotiation.cc:769] Sending connection context
0620 11:42:53.030380 (+ 79us) client_negotiation.cc:240] Negotiation successful
0620 11:42:53.030487 (+ 107us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":133,"thread_start_us":82,"threads_started":1}
Jun 20 11:42:53 dist-test-slave-cjc6 krb5kdc[13130](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781955772, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.6.234.253@KRBTEST.COM
I20260620 11:42:53.031374 13707 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.6.234.253" port: 41509 } has no permanent_uuid. Determining permanent_uuid...
I20260620 11:42:53.034382 13711 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 11:42:53.025945 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38300 (local address 127.6.234.253:41509)
0620 11:42:53.026098 (+ 153us) server_negotiation.cc:206] Beginning negotiation
0620 11:42:53.026103 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 11:42:53.026122 (+ 19us) server_negotiation.cc:407] Connection header received
0620 11:42:53.026143 (+ 21us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 11:42:53.026144 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 11:42:53.026163 (+ 19us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 11:42:53.026245 (+ 82us) server_negotiation.cc:226] Negotiated authn=SASL
0620 11:42:53.026783 (+ 538us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 11:42:53.027728 (+ 945us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 11:42:53.028818 (+ 1090us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 11:42:53.029141 (+ 323us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 11:42:53.029214 (+ 73us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 11:42:53.031827 (+ 2613us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 11:42:53.031835 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 11:42:53.031835 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 11:42:53.031842 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0620 11:42:53.033163 (+ 1321us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 11:42:53.033526 (+ 363us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 11:42:53.033529 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 11:42:53.033529 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 11:42:53.033587 (+ 58us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 11:42:53.033800 (+ 213us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 11:42:53.033801 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 11:42:53.033801 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 11:42:53.033941 (+ 140us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 11:42:53.034126 (+ 185us) server_negotiation.cc:1035] Waiting for connection context
0620 11:42:53.034198 (+ 72us) server_negotiation.cc:299] Negotiation successful
0620 11:42:53.034277 (+ 79us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":123,"thread_start_us":78,"threads_started":1}
Jun 20 11:42:53 dist-test-slave-cjc6 krb5kdc[13130](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781955772, etypes {rep=17 tkt=17 ses=17}, oryx/127.6.234.253@KRBTEST.COM for oryx/127.6.234.253@KRBTEST.COM
I20260620 11:42:53.038751 13709 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 11:42:53.031620 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.234.253:41509 (local address 127.0.0.1:38306)
0620 11:42:53.031843 (+ 223us) negotiation.cc:107] Waiting for socket to connect
0620 11:42:53.031854 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0620 11:42:53.031903 (+ 49us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 11:42:53.032018 (+ 115us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 11:42:53.032233 (+ 215us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 11:42:53.032358 (+ 125us) client_negotiation.cc:189] Negotiated authn=SASL
0620 11:42:53.032401 (+ 43us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 11:42:53.032402 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 11:42:53.033250 (+ 848us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 11:42:53.033251 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 11:42:53.033804 (+ 553us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 11:42:53.033806 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 11:42:53.034174 (+ 368us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 11:42:53.034175 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 11:42:53.034237 (+ 62us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 11:42:53.034636 (+ 399us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 11:42:53.034641 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0620 11:42:53.035858 (+ 1217us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 11:42:53.037960 (+ 2102us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 11:42:53.037961 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 11:42:53.037961 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0620 11:42:53.038110 (+ 149us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 11:42:53.038282 (+ 172us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 11:42:53.038282 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 11:42:53.038283 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0620 11:42:53.038330 (+ 47us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 11:42:53.038512 (+ 182us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 11:42:53.038512 (+ 0us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 11:42:53.038543 (+ 31us) client_negotiation.cc:769] Sending connection context
0620 11:42:53.038591 (+ 48us) client_negotiation.cc:240] Negotiation successful
0620 11:42:53.038664 (+ 73us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":208}
I20260620 11:42:53.038889 13712 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 11:42:53.031694 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38306 (local address 127.6.234.253:41509)
0620 11:42:53.031852 (+ 158us) server_negotiation.cc:206] Beginning negotiation
0620 11:42:53.031854 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 11:42:53.031888 (+ 34us) server_negotiation.cc:407] Connection header received
0620 11:42:53.031955 (+ 67us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 11:42:53.031956 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 11:42:53.031964 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 11:42:53.032018 (+ 54us) server_negotiation.cc:226] Negotiated authn=SASL
0620 11:42:53.032701 (+ 683us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 11:42:53.033157 (+ 456us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 11:42:53.033891 (+ 734us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 11:42:53.034094 (+ 203us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 11:42:53.034175 (+ 81us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 11:42:53.035939 (+ 1764us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 11:42:53.035941 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 11:42:53.035941 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 11:42:53.035946 (+ 5us) server_negotiation.cc:922] calling sasl_server_start()
0620 11:42:53.037894 (+ 1948us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 11:42:53.038181 (+ 287us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 11:42:53.038181 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 11:42:53.038182 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 11:42:53.038228 (+ 46us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 11:42:53.038368 (+ 140us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 11:42:53.038369 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 11:42:53.038369 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 11:42:53.038452 (+ 83us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 11:42:53.038494 (+ 42us) server_negotiation.cc:1035] Waiting for connection context
0620 11:42:53.038605 (+ 111us) server_negotiation.cc:299] Negotiation successful
0620 11:42:53.038717 (+ 112us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":45,"server-negotiator.queue_time_us":141,"thread_start_us":116,"threads_started":1}
I20260620 11:42:53.040140 13707 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 4312e033271940b593bea0332e185bff: Bootstrap starting.
I20260620 11:42:53.040753 13707 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 4312e033271940b593bea0332e185bff: Neither blocks nor log segments found. Creating new log.
I20260620 11:42:53.041033 13707 log.cc:824] T 00000000000000000000000000000000 P 4312e033271940b593bea0332e185bff: Log is configured to *not* fsync() on all Append() calls
I20260620 11:42:53.041872 13707 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 4312e033271940b593bea0332e185bff: No bootstrap required, opened a new log
I20260620 11:42:53.043238 13707 raft_consensus.cc:348] T 00000000000000000000000000000000 P 4312e033271940b593bea0332e185bff [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cc8526b37e1a4e17aa4785a4a8f15156" member_type: VOTER last_known_addr { host: "127.6.234.254" port: 39029 } } peers { permanent_uuid: "4312e033271940b593bea0332e185bff" member_type: VOTER last_known_addr { host: "127.6.234.253" port: 41509 } }
I20260620 11:42:53.043357 13707 raft_consensus.cc:374] T 00000000000000000000000000000000 P 4312e033271940b593bea0332e185bff [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 11:42:53.043380 13707 raft_consensus.cc:729] T 00000000000000000000000000000000 P 4312e033271940b593bea0332e185bff [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4312e033271940b593bea0332e185bff, State: Initialized, Role: FOLLOWER
I20260620 11:42:53.043529 13707 consensus_queue.cc:260] T 00000000000000000000000000000000 P 4312e033271940b593bea0332e185bff [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cc8526b37e1a4e17aa4785a4a8f15156" member_type: VOTER last_known_addr { host: "127.6.234.254" port: 39029 } } peers { permanent_uuid: "4312e033271940b593bea0332e185bff" member_type: VOTER last_known_addr { host: "127.6.234.253" port: 41509 } }
I20260620 11:42:53.043826 13707 sys_catalog.cc:564] T 00000000000000000000000000000000 P 4312e033271940b593bea0332e185bff [sys.catalog]: configured and running, proceeding with master startup.
I20260620 11:42:53.043810 13714 sys_catalog.cc:455] T 00000000000000000000000000000000 P 4312e033271940b593bea0332e185bff [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cc8526b37e1a4e17aa4785a4a8f15156" member_type: VOTER last_known_addr { host: "127.6.234.254" port: 39029 } } peers { permanent_uuid: "4312e033271940b593bea0332e185bff" member_type: VOTER last_known_addr { host: "127.6.234.253" port: 41509 } } }
I20260620 11:42:53.043939 13714 sys_catalog.cc:458] T 00000000000000000000000000000000 P 4312e033271940b593bea0332e185bff [sys.catalog]: This master's current role is: FOLLOWER
W20260620 11:42:53.045403 13725 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 4312e033271940b593bea0332e185bff: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260620 11:42:53.045470 13725 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260620 11:42:53.047068 13639 master_runner.cc:428] Detected that this master 4312e033271940b593bea0332e185bff is joining an existing cluster
I20260620 11:42:53.047107 13639 master_runner.cc:432] Initiating AddMaster RPC to add 127.6.234.253:41509
I20260620 11:42:53.054906 13712 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 11:42:53.048252 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38308 (local address 127.6.234.253:41509)
0620 11:42:53.048282 (+ 30us) server_negotiation.cc:206] Beginning negotiation
0620 11:42:53.048284 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 11:42:53.048426 (+ 142us) server_negotiation.cc:407] Connection header received
0620 11:42:53.048507 (+ 81us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 11:42:53.048508 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 11:42:53.048517 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 11:42:53.048567 (+ 50us) server_negotiation.cc:226] Negotiated authn=SASL
0620 11:42:53.048873 (+ 306us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 11:42:53.049353 (+ 480us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 11:42:53.050389 (+ 1036us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 11:42:53.050703 (+ 314us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 11:42:53.050782 (+ 79us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 11:42:53.051937 (+ 1155us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 11:42:53.051939 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 11:42:53.051939 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 11:42:53.051945 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0620 11:42:53.053687 (+ 1742us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 11:42:53.054100 (+ 413us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 11:42:53.054101 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 11:42:53.054102 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 11:42:53.054172 (+ 70us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 11:42:53.054394 (+ 222us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 11:42:53.054395 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 11:42:53.054396 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 11:42:53.054510 (+ 114us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 11:42:53.054550 (+ 40us) server_negotiation.cc:1035] Waiting for connection context
0620 11:42:53.054729 (+ 179us) server_negotiation.cc:299] Negotiation successful
0620 11:42:53.054802 (+ 73us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":16}
I20260620 11:42:53.055266 13732 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 11:42:53.048071 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.234.253:41509 (local address 127.0.0.1:38308)
0620 11:42:53.048360 (+ 289us) negotiation.cc:107] Waiting for socket to connect
0620 11:42:53.048372 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0620 11:42:53.048441 (+ 69us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 11:42:53.048582 (+ 141us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 11:42:53.048582 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 11:42:53.048749 (+ 167us) client_negotiation.cc:189] Negotiated authn=SASL
0620 11:42:53.048820 (+ 71us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 11:42:53.048821 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 11:42:53.049433 (+ 612us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 11:42:53.049434 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 11:42:53.050300 (+ 866us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 11:42:53.050302 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 11:42:53.050819 (+ 517us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 11:42:53.050821 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 11:42:53.050887 (+ 66us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 11:42:53.051356 (+ 469us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 11:42:53.051361 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0620 11:42:53.051834 (+ 473us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 11:42:53.053790 (+ 1956us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 11:42:53.053791 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 11:42:53.053792 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0620 11:42:53.054019 (+ 227us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 11:42:53.054243 (+ 224us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 11:42:53.054244 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 11:42:53.054245 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0620 11:42:53.054315 (+ 70us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 11:42:53.054589 (+ 274us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 11:42:53.054590 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 11:42:53.054632 (+ 42us) client_negotiation.cc:769] Sending connection context
0620 11:42:53.055081 (+ 449us) client_negotiation.cc:240] Negotiation successful
0620 11:42:53.055171 (+ 90us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":276,"thread_start_us":97,"threads_started":1}
I20260620 11:42:53.055346 13731 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 11:42:53.048056 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.234.254:39029 (local address 127.0.0.1:51588)
0620 11:42:53.048217 (+ 161us) negotiation.cc:107] Waiting for socket to connect
0620 11:42:53.048227 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0620 11:42:53.048292 (+ 65us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 11:42:53.048435 (+ 143us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 11:42:53.048436 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 11:42:53.048662 (+ 226us) client_negotiation.cc:189] Negotiated authn=SASL
0620 11:42:53.048744 (+ 82us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 11:42:53.048745 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 11:42:53.049678 (+ 933us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 11:42:53.049679 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 11:42:53.050474 (+ 795us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 11:42:53.050476 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 11:42:53.050867 (+ 391us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 11:42:53.050868 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 11:42:53.050940 (+ 72us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 11:42:53.051552 (+ 612us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 11:42:53.051557 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0620 11:42:53.052018 (+ 461us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 11:42:53.054083 (+ 2065us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 11:42:53.054083 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 11:42:53.054084 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0620 11:42:53.054314 (+ 230us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 11:42:53.054512 (+ 198us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 11:42:53.054513 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 11:42:53.054513 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0620 11:42:53.054589 (+ 76us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 11:42:53.054847 (+ 258us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 11:42:53.054848 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 11:42:53.054887 (+ 39us) client_negotiation.cc:769] Sending connection context
0620 11:42:53.054970 (+ 83us) client_negotiation.cc:240] Negotiation successful
0620 11:42:53.055064 (+ 94us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":140,"thread_start_us":116,"threads_started":1}
I20260620 11:42:53.055689 13215 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 11:42:53.048099 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51588 (local address 127.6.234.254:39029)
0620 11:42:53.048143 (+ 44us) server_negotiation.cc:206] Beginning negotiation
0620 11:42:53.048145 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 11:42:53.048300 (+ 155us) server_negotiation.cc:407] Connection header received
0620 11:42:53.048349 (+ 49us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 11:42:53.048350 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 11:42:53.048361 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 11:42:53.048416 (+ 55us) server_negotiation.cc:226] Negotiated authn=SASL
0620 11:42:53.048827 (+ 411us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 11:42:53.049596 (+ 769us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 11:42:53.050562 (+ 966us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 11:42:53.050797 (+ 235us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 11:42:53.050869 (+ 72us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 11:42:53.052107 (+ 1238us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 11:42:53.052108 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 11:42:53.052109 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 11:42:53.052115 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0620 11:42:53.053973 (+ 1858us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 11:42:53.054384 (+ 411us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 11:42:53.054385 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 11:42:53.054386 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 11:42:53.054438 (+ 52us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 11:42:53.054654 (+ 216us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 11:42:53.054655 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 11:42:53.054656 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 11:42:53.054768 (+ 112us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 11:42:53.054843 (+ 75us) server_negotiation.cc:1035] Waiting for connection context
0620 11:42:53.055189 (+ 346us) server_negotiation.cc:299] Negotiation successful
0620 11:42:53.055271 (+ 82us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":22}
Jun 20 11:42:53 dist-test-slave-cjc6 krb5kdc[13130](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781955772, etypes {rep=17 tkt=17 ses=17}, oryx/127.6.234.254@KRBTEST.COM for oryx/127.6.234.253@KRBTEST.COM
I20260620 11:42:53.064395 13712 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 11:42:53.057763 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38314 (local address 127.6.234.253:41509)
0620 11:42:53.057809 (+ 46us) server_negotiation.cc:206] Beginning negotiation
0620 11:42:53.057811 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 11:42:53.057932 (+ 121us) server_negotiation.cc:407] Connection header received
0620 11:42:53.058020 (+ 88us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 11:42:53.058021 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 11:42:53.058031 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 11:42:53.058086 (+ 55us) server_negotiation.cc:226] Negotiated authn=SASL
0620 11:42:53.058489 (+ 403us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 11:42:53.058943 (+ 454us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 11:42:53.059706 (+ 763us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 11:42:53.059909 (+ 203us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 11:42:53.060003 (+ 94us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 11:42:53.061941 (+ 1938us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 11:42:53.061942 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 11:42:53.061943 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 11:42:53.061949 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0620 11:42:53.063174 (+ 1225us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 11:42:53.063569 (+ 395us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 11:42:53.063570 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 11:42:53.063571 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 11:42:53.063629 (+ 58us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 11:42:53.063826 (+ 197us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 11:42:53.063827 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 11:42:53.063827 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 11:42:53.063934 (+ 107us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 11:42:53.064009 (+ 75us) server_negotiation.cc:1035] Waiting for connection context
0620 11:42:53.064233 (+ 224us) server_negotiation.cc:299] Negotiation successful
0620 11:42:53.064307 (+ 74us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":21}
I20260620 11:42:53.064411 13733 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 11:42:53.057689 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.234.253:41509 (local address 127.0.0.1:38314)
0620 11:42:53.057824 (+ 135us) negotiation.cc:107] Waiting for socket to connect
0620 11:42:53.057862 (+ 38us) client_negotiation.cc:174] Beginning negotiation
0620 11:42:53.057957 (+ 95us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 11:42:53.058092 (+ 135us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 11:42:53.058095 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 11:42:53.058362 (+ 267us) client_negotiation.cc:189] Negotiated authn=SASL
0620 11:42:53.058407 (+ 45us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 11:42:53.058409 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 11:42:53.059030 (+ 621us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 11:42:53.059031 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 11:42:53.059611 (+ 580us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 11:42:53.059613 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 11:42:53.059988 (+ 375us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 11:42:53.059989 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 11:42:53.060088 (+ 99us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 11:42:53.060508 (+ 420us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 11:42:53.060512 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0620 11:42:53.061844 (+ 1332us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 11:42:53.063273 (+ 1429us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 11:42:53.063280 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 11:42:53.063282 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0620 11:42:53.063482 (+ 200us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 11:42:53.063699 (+ 217us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 11:42:53.063700 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 11:42:53.063700 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0620 11:42:53.063762 (+ 62us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 11:42:53.064002 (+ 240us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 11:42:53.064010 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 11:42:53.064092 (+ 82us) client_negotiation.cc:769] Sending connection context
0620 11:42:53.064190 (+ 98us) client_negotiation.cc:240] Negotiation successful
0620 11:42:53.064320 (+ 130us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":106,"thread_start_us":62,"threads_started":1}
I20260620 11:42:53.065197 13175 catalog_manager.cc:6979] Initiating ChangeConfig request to add master 127.6.234.253:41509: tablet_id: "00000000000000000000000000000000"
type: ADD_PEER
server {
permanent_uuid: "4312e033271940b593bea0332e185bff"
member_type: NON_VOTER
last_known_addr {
host: "127.6.234.253"
port: 41509
}
attrs {
promote: true
}
}
dest_uuid: "cc8526b37e1a4e17aa4785a4a8f15156"
cas_config_opid_index: -1
I20260620 11:42:53.065516 13175 consensus_queue.cc:237] T 00000000000000000000000000000000 P cc8526b37e1a4e17aa4785a4a8f15156 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "cc8526b37e1a4e17aa4785a4a8f15156" member_type: VOTER last_known_addr { host: "127.6.234.254" port: 39029 } } peers { permanent_uuid: "4312e033271940b593bea0332e185bff" member_type: NON_VOTER last_known_addr { host: "127.6.234.253" port: 41509 } attrs { promote: true } }
I20260620 11:42:53.066012 13735 raft_consensus.cc:2932] T 00000000000000000000000000000000 P cc8526b37e1a4e17aa4785a4a8f15156 [term 1 LEADER]: Committing config change with OpId 1.5: config changed from index -1 to 5, NON_VOTER 4312e033271940b593bea0332e185bff (127.6.234.253) added. New config: { opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "cc8526b37e1a4e17aa4785a4a8f15156" member_type: VOTER last_known_addr { host: "127.6.234.254" port: 39029 } } peers { permanent_uuid: "4312e033271940b593bea0332e185bff" member_type: NON_VOTER last_known_addr { host: "127.6.234.253" port: 41509 } attrs { promote: true } } }
I20260620 11:42:53.066367 13681 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 4312e033271940b593bea0332e185bff [term 0 FOLLOWER]: Advancing to term 1
I20260620 11:42:53.066948 13735 catalog_manager.cc:6983] Successfully completed master ChangeConfig request to add master 127.6.234.253:41509
I20260620 11:42:53.067148 13639 master.cc:537] Master@127.6.234.253:41509 shutting down...
I20260620 11:42:53.067277 13681 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 4312e033271940b593bea0332e185bff [term 1 FOLLOWER]: Refusing update from remote peer cc8526b37e1a4e17aa4785a4a8f15156: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20260620 11:42:53.067193 13734 sys_catalog.cc:455] T 00000000000000000000000000000000 P cc8526b37e1a4e17aa4785a4a8f15156 [sys.catalog]: SysCatalogTable state changed. Reason: Config change replication complete. Latest consensus state: current_term: 1 leader_uuid: "cc8526b37e1a4e17aa4785a4a8f15156" committed_config { opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "cc8526b37e1a4e17aa4785a4a8f15156" member_type: VOTER last_known_addr { host: "127.6.234.254" port: 39029 } } peers { permanent_uuid: "4312e033271940b593bea0332e185bff" member_type: NON_VOTER last_known_addr { host: "127.6.234.253" port: 41509 } attrs { promote: true } } }
I20260620 11:42:53.067421 13734 sys_catalog.cc:458] T 00000000000000000000000000000000 P cc8526b37e1a4e17aa4785a4a8f15156 [sys.catalog]: This master's current role is: LEADER
I20260620 11:42:53.067679 13734 consensus_queue.cc:1035] T 00000000000000000000000000000000 P cc8526b37e1a4e17aa4785a4a8f15156 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4312e033271940b593bea0332e185bff" member_type: NON_VOTER last_known_addr { host: "127.6.234.253" port: 41509 } attrs { promote: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 0, Time since last communication: 0.000s
W20260620 11:42:53.068336 13161 proxy.cc:239] Call had error, refreshing address and retrying: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on Master
W20260620 11:42:53.068918 13161 consensus_peers.cc:487] T 00000000000000000000000000000000 P cc8526b37e1a4e17aa4785a4a8f15156 -> Peer 4312e033271940b593bea0332e185bff (127.6.234.253:41509): Couldn't send request to peer 4312e033271940b593bea0332e185bff. Status: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on Master. This is attempt 1: this message will repeat every 5th retry.
I20260620 11:42:53.070116 13639 raft_consensus.cc:2229] T 00000000000000000000000000000000 P 4312e033271940b593bea0332e185bff [term 1 FOLLOWER]: Raft consensus shutting down.
I20260620 11:42:53.070210 13639 raft_consensus.cc:2258] T 00000000000000000000000000000000 P 4312e033271940b593bea0332e185bff [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 11:42:53.070262 13639 tablet_replica.cc:330] T 00000000000000000000000000000000 P 4312e033271940b593bea0332e185bff: stopping tablet replica
I20260620 11:42:53.072588 13639 master.cc:552] Master@127.6.234.253:41509 shutdown complete.
I20260620 11:42:53.072639 13639 master_runner.cc:305] Clearing existing system tablet
I20260620 11:42:53.072789 13639 fs_manager.cc:359] Metadata directory not provided
I20260620 11:42:53.072829 13639 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 11:42:53.073390 13639 fs_manager.cc:709] Time spent opening directory manager: real 0.000s user 0.001s sys 0.000s
I20260620 11:42:53.073894 13739 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 11:42:53.074015 13639 fs_report.cc:383] FS layout report
--------------------
wal directory:
metadata directory:
1 data directories: /tmp/dist-test-task1JRg7N/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781955746769051-7083-0/minicluster-data/master-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 11:42:53.074070 13639 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.000s
I20260620 11:42:53.074095 13639 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task1JRg7N/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781955746769051-7083-0/minicluster-data/master-1/data,/tmp/dist-test-task1JRg7N/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781955746769051-7083-0/minicluster-data/master-1/wal
uuid: "4312e033271940b593bea0332e185bff"
format_stamp: "Formatted at 2026-06-20 11:42:52 on dist-test-slave-cjc6"
server_key: "e2dc6fee535f92e5e4241117a118701d"
server_key_iv: "d0867c552837f801b2336ce81ba05922"
server_key_version: "encryptionkey@0"
I20260620 11:42:53.074657 13639 ts_tablet_manager.cc:1905] T 00000000000000000000000000000000 P 4312e033271940b593bea0332e185bff: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 11:42:53.075837 13639 ts_tablet_manager.cc:1918] T 00000000000000000000000000000000 P 4312e033271940b593bea0332e185bff: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId unknown
I20260620 11:42:53.075891 13639 log.cc:1196] T 00000000000000000000000000000000 P 4312e033271940b593bea0332e185bff: Deleting WAL directory at /tmp/dist-test-task1JRg7N/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781955746769051-7083-0/minicluster-data/master-1/wal/wals/00000000000000000000000000000000
I20260620 11:42:53.076086 13639 ts_tablet_manager.cc:1939] T 00000000000000000000000000000000 P 4312e033271940b593bea0332e185bff: Deleting consensus metadata
I20260620 11:42:53.076210 13639 master_runner.cc:315] Copying system tablet from 127.6.234.254:39029
I20260620 11:42:53.076786 13639 tablet_copy_client.cc:316] T 00000000000000000000000000000000 P 4312e033271940b593bea0332e185bff: tablet copy: Beginning tablet copy session from remote peer at address 127.6.234.254:39029
I20260620 11:42:53.082695 13215 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 11:42:53.077143 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51594 (local address 127.6.234.254:39029)
0620 11:42:53.077184 (+ 41us) server_negotiation.cc:206] Beginning negotiation
0620 11:42:53.077186 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 11:42:53.077313 (+ 127us) server_negotiation.cc:407] Connection header received
0620 11:42:53.077383 (+ 70us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 11:42:53.077384 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 11:42:53.077393 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 11:42:53.077449 (+ 56us) server_negotiation.cc:226] Negotiated authn=SASL
0620 11:42:53.077790 (+ 341us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 11:42:53.078277 (+ 487us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 11:42:53.079018 (+ 741us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 11:42:53.079223 (+ 205us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 11:42:53.079311 (+ 88us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 11:42:53.080225 (+ 914us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 11:42:53.080227 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 11:42:53.080227 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 11:42:53.080233 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0620 11:42:53.081460 (+ 1227us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 11:42:53.081910 (+ 450us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 11:42:53.081912 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 11:42:53.081912 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 11:42:53.081970 (+ 58us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 11:42:53.082181 (+ 211us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 11:42:53.082181 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 11:42:53.082182 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 11:42:53.082286 (+ 104us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 11:42:53.082361 (+ 75us) server_negotiation.cc:1035] Waiting for connection context
0620 11:42:53.082532 (+ 171us) server_negotiation.cc:299] Negotiation successful
0620 11:42:53.082606 (+ 74us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":22}
I20260620 11:42:53.082711 13745 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 11:42:53.077090 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.234.254:39029 (local address 127.0.0.1:51594)
0620 11:42:53.077231 (+ 141us) negotiation.cc:107] Waiting for socket to connect
0620 11:42:53.077248 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0620 11:42:53.077322 (+ 74us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 11:42:53.077454 (+ 132us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 11:42:53.077455 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 11:42:53.077630 (+ 175us) client_negotiation.cc:189] Negotiated authn=SASL
0620 11:42:53.077681 (+ 51us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 11:42:53.077683 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 11:42:53.078372 (+ 689us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 11:42:53.078373 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 11:42:53.078920 (+ 547us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 11:42:53.078922 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 11:42:53.079306 (+ 384us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 11:42:53.079307 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 11:42:53.079372 (+ 65us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 11:42:53.079846 (+ 474us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 11:42:53.079850 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0620 11:42:53.080129 (+ 279us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 11:42:53.081554 (+ 1425us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 11:42:53.081555 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 11:42:53.081556 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0620 11:42:53.081825 (+ 269us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 11:42:53.082045 (+ 220us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 11:42:53.082046 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 11:42:53.082046 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0620 11:42:53.082110 (+ 64us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 11:42:53.082365 (+ 255us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 11:42:53.082366 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 11:42:53.082422 (+ 56us) client_negotiation.cc:769] Sending connection context
0620 11:42:53.082505 (+ 83us) client_negotiation.cc:240] Negotiation successful
0620 11:42:53.082608 (+ 103us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":122,"thread_start_us":66,"threads_started":1}
I20260620 11:42:53.082974 13195 tablet_copy_service.cc:140] P cc8526b37e1a4e17aa4785a4a8f15156: Received BeginTabletCopySession request for tablet 00000000000000000000000000000000 from peer 4312e033271940b593bea0332e185bff ({username='oryx', principal='oryx/127.6.234.253@KRBTEST.COM'} at 127.0.0.1:51594)
I20260620 11:42:53.083089 13195 tablet_copy_service.cc:161] P cc8526b37e1a4e17aa4785a4a8f15156: Beginning new tablet copy session on tablet 00000000000000000000000000000000 from peer 4312e033271940b593bea0332e185bff at {username='oryx', principal='oryx/127.6.234.253@KRBTEST.COM'} at 127.0.0.1:51594: session id = 4312e033271940b593bea0332e185bff-00000000000000000000000000000000
I20260620 11:42:53.083673 13195 tablet_copy_source_session.cc:205] T 00000000000000000000000000000000 P cc8526b37e1a4e17aa4785a4a8f15156: Tablet Copy: opened 0 blocks and 1 log segments
I20260620 11:42:53.084251 13639 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total
I20260620 11:42:53.085140 13639 tablet_copy_client.cc:791] T 00000000000000000000000000000000 P 4312e033271940b593bea0332e185bff: tablet copy: Starting download of 0 data blocks...
I20260620 11:42:53.085310 13639 tablet_copy_client.cc:655] T 00000000000000000000000000000000 P 4312e033271940b593bea0332e185bff: tablet copy: Starting download of 1 WAL segments...
I20260620 11:42:53.086125 13639 tablet_copy_client.cc:531] T 00000000000000000000000000000000 P 4312e033271940b593bea0332e185bff: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20260620 11:42:53.086905 13195 tablet_copy_service.cc:342] P cc8526b37e1a4e17aa4785a4a8f15156: Request end of tablet copy session 4312e033271940b593bea0332e185bff-00000000000000000000000000000000 received from {username='oryx', principal='oryx/127.6.234.253@KRBTEST.COM'} at 127.0.0.1:51594
I20260620 11:42:53.086961 13195 tablet_copy_service.cc:433] P cc8526b37e1a4e17aa4785a4a8f15156: ending tablet copy session 4312e033271940b593bea0332e185bff-00000000000000000000000000000000 on tablet 00000000000000000000000000000000 with peer 4312e033271940b593bea0332e185bff
I20260620 11:42:53.089342 13639 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 11:42:53.090543 13747 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 11:42:53.090588 13746 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 11:42:53.090781 13639 server_base.cc:1034] running on GCE node
W20260620 11:42:53.095664 13749 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 11:42:53.095845 13639 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 11:42:53.096015 13639 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 11:42:53.097180 13639 hybrid_clock.cc:648] HybridClock initialized: now 1781955773097172 us; error 24 us; skew 500 ppm
Jun 20 11:42:53 dist-test-slave-cjc6 krb5kdc[13130](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781955773, etypes {rep=17 tkt=17 ses=17}, oryx/127.6.234.253@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 11:42:53.098507 13639 init.cc:377] Logged in from keytab as oryx/127.6.234.253@KRBTEST.COM (short username oryx)
I20260620 11:42:53.098809 13639 webserver.cc:460] Webserver started at http://127.6.234.253:46781/ using document root <none> and password file <none>
I20260620 11:42:53.098933 13639 fs_manager.cc:359] Metadata directory not provided
I20260620 11:42:53.098961 13639 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 11:42:53.099825 13639 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.001s sys 0.000s
I20260620 11:42:53.100425 13756 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 11:42:53.100548 13639 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260620 11:42:53.100601 13639 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task1JRg7N/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781955746769051-7083-0/minicluster-data/master-1/data,/tmp/dist-test-task1JRg7N/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781955746769051-7083-0/minicluster-data/master-1/wal
uuid: "4312e033271940b593bea0332e185bff"
format_stamp: "Formatted at 2026-06-20 11:42:52 on dist-test-slave-cjc6"
server_key: "e2dc6fee535f92e5e4241117a118701d"
server_key_iv: "d0867c552837f801b2336ce81ba05922"
server_key_version: "encryptionkey@0"
I20260620 11:42:53.100651 13639 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task1JRg7N/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781955746769051-7083-0/minicluster-data/master-1/wal
metadata directory: /tmp/dist-test-task1JRg7N/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781955746769051-7083-0/minicluster-data/master-1/wal
1 data directories: /tmp/dist-test-task1JRg7N/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781955746769051-7083-0/minicluster-data/master-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 11:42:53.120057 13639 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 11:42:53.121598 13639 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 11:42:53.124693 13639 rpc_server.cc:307] RPC server started. Bound to: 127.6.234.253:41509
I20260620 11:42:53.124851 13808 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.6.234.253:41509 every 8 connection(s)
I20260620 11:42:53.125051 13639 server_base.cc:1166] Dumped server information to /tmp/dist-test-task1JRg7N/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781955746769051-7083-0/minicluster-data/master-1/data/info.pb
I20260620 11:42:53.125804 13809 sys_catalog.cc:263] Verifying existing consensus state
I20260620 11:42:53.126227 13809 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 4312e033271940b593bea0332e185bff: Bootstrap starting.
I20260620 11:42:53.128383 13809 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 4312e033271940b593bea0332e185bff: Bootstrap replayed 1/1 log segments. Stats: ops{read=5 overwritten=0 applied=5 ignored=0} inserts{seen=3 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260620 11:42:53.128613 13809 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 4312e033271940b593bea0332e185bff: Bootstrap complete.
I20260620 11:42:53.128837 13809 raft_consensus.cc:348] T 00000000000000000000000000000000 P 4312e033271940b593bea0332e185bff [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "cc8526b37e1a4e17aa4785a4a8f15156" member_type: VOTER last_known_addr { host: "127.6.234.254" port: 39029 } } peers { permanent_uuid: "4312e033271940b593bea0332e185bff" member_type: NON_VOTER last_known_addr { host: "127.6.234.253" port: 41509 } attrs { promote: true } }
I20260620 11:42:53.128906 13809 raft_consensus.cc:729] T 00000000000000000000000000000000 P 4312e033271940b593bea0332e185bff [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 4312e033271940b593bea0332e185bff, State: Initialized, Role: LEARNER
I20260620 11:42:53.128932 13809 consensus_queue.cc:260] T 00000000000000000000000000000000 P 4312e033271940b593bea0332e185bff [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 5, Last appended: 1.5, Last appended by leader: 5, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "cc8526b37e1a4e17aa4785a4a8f15156" member_type: VOTER last_known_addr { host: "127.6.234.254" port: 39029 } } peers { permanent_uuid: "4312e033271940b593bea0332e185bff" member_type: NON_VOTER last_known_addr { host: "127.6.234.253" port: 41509 } attrs { promote: true } }
I20260620 11:42:53.129117 13809 sys_catalog.cc:564] T 00000000000000000000000000000000 P 4312e033271940b593bea0332e185bff [sys.catalog]: configured and running, proceeding with master startup.
I20260620 11:42:53.129118 13812 sys_catalog.cc:455] T 00000000000000000000000000000000 P 4312e033271940b593bea0332e185bff [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 committed_config { opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "cc8526b37e1a4e17aa4785a4a8f15156" member_type: VOTER last_known_addr { host: "127.6.234.254" port: 39029 } } peers { permanent_uuid: "4312e033271940b593bea0332e185bff" member_type: NON_VOTER last_known_addr { host: "127.6.234.253" port: 41509 } attrs { promote: true } } }
I20260620 11:42:53.129243 13812 sys_catalog.cc:458] T 00000000000000000000000000000000 P 4312e033271940b593bea0332e185bff [sys.catalog]: This master's current role is: LEARNER
I20260620 11:42:53.130612 13823 catalog_manager.cc:1259] Loaded cluster ID: 62ddcf685f6646508efb7ea8accaf0cf
I20260620 11:42:53.130683 13823 catalog_manager.cc:1552] T 00000000000000000000000000000000 P 4312e033271940b593bea0332e185bff: loading cluster ID for follower catalog manager: success
I20260620 11:42:53.131608 13823 catalog_manager.cc:1574] T 00000000000000000000000000000000 P 4312e033271940b593bea0332e185bff: acquiring CA information for follower catalog manager: success
I20260620 11:42:53.131886 13823 catalog_manager.cc:1602] T 00000000000000000000000000000000 P 4312e033271940b593bea0332e185bff: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20260620 11:42:53.143141 13824 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 11:42:53.136164 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38336 (local address 127.6.234.253:41509)
0620 11:42:53.136267 (+ 103us) server_negotiation.cc:206] Beginning negotiation
0620 11:42:53.136269 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 11:42:53.136279 (+ 10us) server_negotiation.cc:407] Connection header received
0620 11:42:53.136304 (+ 25us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 11:42:53.136304 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 11:42:53.136315 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 11:42:53.136405 (+ 90us) server_negotiation.cc:226] Negotiated authn=SASL
0620 11:42:53.136934 (+ 529us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 11:42:53.137978 (+ 1044us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 11:42:53.138730 (+ 752us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 11:42:53.139047 (+ 317us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 11:42:53.139127 (+ 80us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 11:42:53.140336 (+ 1209us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 11:42:53.140337 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 11:42:53.140338 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 11:42:53.140344 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0620 11:42:53.141981 (+ 1637us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 11:42:53.142338 (+ 357us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 11:42:53.142339 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 11:42:53.142340 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 11:42:53.142405 (+ 65us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 11:42:53.142591 (+ 186us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 11:42:53.142592 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 11:42:53.142592 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 11:42:53.142732 (+ 140us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 11:42:53.142843 (+ 111us) server_negotiation.cc:1035] Waiting for connection context
0620 11:42:53.142950 (+ 107us) server_negotiation.cc:299] Negotiation successful
0620 11:42:53.143032 (+ 82us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":86,"thread_start_us":48,"threads_started":1}
WARNING: no policy specified for oryx/127.6.234.252@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.6.234.252@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.6.234.252 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task1JRg7N/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781955746769051-7083-0/krb5kdc/oryx_127.6.234.252.keytab.
Entry for principal oryx/127.6.234.252 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task1JRg7N/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781955746769051-7083-0/krb5kdc/oryx_127.6.234.252.keytab.
I20260620 11:42:53.168272 7083 external_mini_cluster.cc:1326] Running /tmp/dist-test-task1JRg7N/build/release/bin/kudu
/tmp/dist-test-task1JRg7N/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task1JRg7N/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781955746769051-7083-0/minicluster-data/master-2/wal
--fs_data_dirs=/tmp/dist-test-task1JRg7N/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781955746769051-7083-0/minicluster-data/master-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task1JRg7N/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781955746769051-7083-0/minicluster-data/master-2/logs
--server_dump_info_path=/tmp/dist-test-task1JRg7N/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781955746769051-7083-0/minicluster-data/master-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.6.234.252:39551
--webserver_interface=127.6.234.252
--webserver_port=0
--keytab_file=/tmp/dist-test-task1JRg7N/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781955746769051-7083-0/krb5kdc/oryx_127.6.234.252.keytab
--principal=oryx/127.6.234.252
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.6.234.212:38681
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.6.234.254:39029,127.6.234.253:41509,127.6.234.252:39551
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task1JRg7N/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781955746769051-7083-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task1JRg7N/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781955746769051-7083-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task1JRg7N/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781955746769051-7083-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 11:42:53.249308 13827 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 11:42:53.249543 13827 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 11:42:53.249581 13827 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 11:42:53.250787 13827 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 11:42:53.250859 13827 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 11:42:53.250885 13827 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 11:42:53.250916 13827 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 11:42:53.250942 13827 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 11:42:53.252511 13827 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.6.234.212:38681
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task1JRg7N/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781955746769051-7083-0/minicluster-data/master-2/data
--fs_wal_dir=/tmp/dist-test-task1JRg7N/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781955746769051-7083-0/minicluster-data/master-2/wal
--ipki_ca_key_size=768
--master_addresses=127.6.234.254:39029,127.6.234.253:41509,127.6.234.252:39551
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task1JRg7N/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781955746769051-7083-0/krb5kdc/oryx_127.6.234.252.keytab
--principal=oryx/127.6.234.252
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.6.234.252:39551
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task1JRg7N/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781955746769051-7083-0/minicluster-data/master-2/data/info.pb
--webserver_interface=127.6.234.252
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.13827
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task1JRg7N/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781955746769051-7083-0/minicluster-data/master-2/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 20 Jun 2026 10:32:37 UTC on ccycloud.quasar-bgyofv.root.comops.site
build id 4457173
I20260620 11:42:53.252784 13827 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 11:42:53.253005 13827 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 11:42:53.257678 13835 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 11:42:53.257735 13832 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 11:42:53.257903 13827 server_base.cc:1034] running on GCE node
W20260620 11:42:53.257936 13833 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 11:42:53.258164 13827 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 11:42:53.258455 13827 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 11:42:53.259610 13827 hybrid_clock.cc:648] HybridClock initialized: now 1781955773259608 us; error 35 us; skew 500 ppm
Jun 20 11:42:53 dist-test-slave-cjc6 krb5kdc[13130](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781955773, etypes {rep=17 tkt=17 ses=17}, oryx/127.6.234.252@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 11:42:53.266886 13827 init.cc:377] Logged in from keytab as oryx/127.6.234.252@KRBTEST.COM (short username oryx)
I20260620 11:42:53.267379 13827 webserver.cc:460] Webserver started at http://127.6.234.252:33159/ using document root <none> and password file <none>
I20260620 11:42:53.267635 13827 fs_manager.cc:359] Metadata directory not provided
I20260620 11:42:53.267709 13827 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 11:42:53.267843 13827 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 11:42:53.269021 13827 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task1JRg7N/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781955746769051-7083-0/minicluster-data/master-2/data/instance:
uuid: "ea509afd379942ea887648a5de383571"
format_stamp: "Formatted at 2026-06-20 11:42:53 on dist-test-slave-cjc6"
server_key: "b789541cb9267110241b5dc5b2ff9bd5"
server_key_iv: "8841afa5c047fc6f3706f7d306db14e8"
server_key_version: "encryptionkey@0"
I20260620 11:42:53.269412 13827 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task1JRg7N/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781955746769051-7083-0/minicluster-data/master-2/wal/instance:
uuid: "ea509afd379942ea887648a5de383571"
format_stamp: "Formatted at 2026-06-20 11:42:53 on dist-test-slave-cjc6"
server_key: "b789541cb9267110241b5dc5b2ff9bd5"
server_key_iv: "8841afa5c047fc6f3706f7d306db14e8"
server_key_version: "encryptionkey@0"
I20260620 11:42:53.270956 13827 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260620 11:42:53.271718 13842 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 11:42:53.271898 13827 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260620 11:42:53.272004 13827 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task1JRg7N/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781955746769051-7083-0/minicluster-data/master-2/data,/tmp/dist-test-task1JRg7N/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781955746769051-7083-0/minicluster-data/master-2/wal
uuid: "ea509afd379942ea887648a5de383571"
format_stamp: "Formatted at 2026-06-20 11:42:53 on dist-test-slave-cjc6"
server_key: "b789541cb9267110241b5dc5b2ff9bd5"
server_key_iv: "8841afa5c047fc6f3706f7d306db14e8"
server_key_version: "encryptionkey@0"
I20260620 11:42:53.272094 13827 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task1JRg7N/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781955746769051-7083-0/minicluster-data/master-2/wal
metadata directory: /tmp/dist-test-task1JRg7N/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781955746769051-7083-0/minicluster-data/master-2/wal
1 data directories: /tmp/dist-test-task1JRg7N/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781955746769051-7083-0/minicluster-data/master-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 11:42:53.281888 13827 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 11:42:53.284056 13827 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 11:42:53.284229 13827 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 11:42:53.288543 13827 rpc_server.cc:307] RPC server started. Bound to: 127.6.234.252:39551
I20260620 11:42:53.288574 13894 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.6.234.252:39551 every 8 connection(s)
I20260620 11:42:53.288938 13827 server_base.cc:1166] Dumped server information to /tmp/dist-test-task1JRg7N/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781955746769051-7083-0/minicluster-data/master-2/data/info.pb
I20260620 11:42:53.289989 13895 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 11:42:53.291597 13895 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.6.234.254" port: 39029 } has no permanent_uuid. Determining permanent_uuid...
I20260620 11:42:53.293327 7083 external_mini_cluster.cc:1388] Started /tmp/dist-test-task1JRg7N/build/release/bin/kudu as pid 13827
Jun 20 11:42:53 dist-test-slave-cjc6 krb5kdc[13130](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781955773, etypes {rep=17 tkt=17 ses=17}, oryx/127.6.234.252@KRBTEST.COM for oryx/127.6.234.254@KRBTEST.COM
Jun 20 11:42:53 dist-test-slave-cjc6 krb5kdc[13130](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781955772, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.6.234.252@KRBTEST.COM
I20260620 11:42:53.302529 13215 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 11:42:53.292509 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51602 (local address 127.6.234.254:39029)
0620 11:42:53.292556 (+ 47us) server_negotiation.cc:206] Beginning negotiation
0620 11:42:53.292559 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0620 11:42:53.292598 (+ 39us) server_negotiation.cc:407] Connection header received
0620 11:42:53.293096 (+ 498us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 11:42:53.293098 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 11:42:53.293110 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 11:42:53.293161 (+ 51us) server_negotiation.cc:226] Negotiated authn=SASL
0620 11:42:53.294063 (+ 902us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 11:42:53.294984 (+ 921us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 11:42:53.296319 (+ 1335us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 11:42:53.296580 (+ 261us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 11:42:53.296658 (+ 78us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 11:42:53.299329 (+ 2671us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 11:42:53.299331 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 11:42:53.299331 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 11:42:53.299337 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0620 11:42:53.300967 (+ 1630us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 11:42:53.301489 (+ 522us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 11:42:53.301490 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 11:42:53.301490 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 11:42:53.301543 (+ 53us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 11:42:53.301886 (+ 343us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 11:42:53.301887 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 11:42:53.301887 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 11:42:53.302002 (+ 115us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 11:42:53.302057 (+ 55us) server_negotiation.cc:1035] Waiting for connection context
0620 11:42:53.302406 (+ 349us) server_negotiation.cc:299] Negotiation successful
0620 11:42:53.302467 (+ 61us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":21}
I20260620 11:42:53.302799 13897 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 11:42:53.292308 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.234.254:39029 (local address 127.0.0.1:51602)
0620 11:42:53.292481 (+ 173us) negotiation.cc:107] Waiting for socket to connect
0620 11:42:53.292521 (+ 40us) client_negotiation.cc:174] Beginning negotiation
0620 11:42:53.292984 (+ 463us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 11:42:53.293244 (+ 260us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 11:42:53.293247 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 11:42:53.293543 (+ 296us) client_negotiation.cc:189] Negotiated authn=SASL
0620 11:42:53.293783 (+ 240us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 11:42:53.293789 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 11:42:53.295091 (+ 1302us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 11:42:53.295092 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 11:42:53.296241 (+ 1149us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 11:42:53.296243 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 11:42:53.296678 (+ 435us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 11:42:53.296679 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 11:42:53.296775 (+ 96us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 11:42:53.297318 (+ 543us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 11:42:53.297326 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0620 11:42:53.299243 (+ 1917us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 11:42:53.301045 (+ 1802us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 11:42:53.301053 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 11:42:53.301055 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0620 11:42:53.301403 (+ 348us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 11:42:53.301633 (+ 230us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 11:42:53.301635 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 11:42:53.301635 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0620 11:42:53.301810 (+ 175us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 11:42:53.302076 (+ 266us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 11:42:53.302080 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 11:42:53.302256 (+ 176us) client_negotiation.cc:769] Sending connection context
0620 11:42:53.302375 (+ 119us) client_negotiation.cc:240] Negotiation successful
0620 11:42:53.302540 (+ 165us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":126,"thread_start_us":93,"threads_started":1}
I20260620 11:42:53.303352 13895 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.6.234.253" port: 41509 } has no permanent_uuid. Determining permanent_uuid...
I20260620 11:42:53.303762 13898 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 11:42:53.294044 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43390 (local address 127.6.234.252:39551)
0620 11:42:53.294266 (+ 222us) server_negotiation.cc:206] Beginning negotiation
0620 11:42:53.294271 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 11:42:53.294296 (+ 25us) server_negotiation.cc:407] Connection header received
0620 11:42:53.294318 (+ 22us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 11:42:53.294320 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 11:42:53.294339 (+ 19us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 11:42:53.294491 (+ 152us) server_negotiation.cc:226] Negotiated authn=SASL
0620 11:42:53.295231 (+ 740us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 11:42:53.296411 (+ 1180us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 11:42:53.297252 (+ 841us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 11:42:53.297573 (+ 321us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 11:42:53.297659 (+ 86us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 11:42:53.300379 (+ 2720us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 11:42:53.300386 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 11:42:53.300389 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 11:42:53.300401 (+ 12us) server_negotiation.cc:922] calling sasl_server_start()
0620 11:42:53.302417 (+ 2016us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 11:42:53.302854 (+ 437us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 11:42:53.302856 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 11:42:53.302857 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 11:42:53.302918 (+ 61us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 11:42:53.303159 (+ 241us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 11:42:53.303159 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 11:42:53.303160 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 11:42:53.303321 (+ 161us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 11:42:53.303459 (+ 138us) server_negotiation.cc:1035] Waiting for connection context
0620 11:42:53.303552 (+ 93us) server_negotiation.cc:299] Negotiation successful
0620 11:42:53.303629 (+ 77us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":188,"thread_start_us":111,"threads_started":1}
Jun 20 11:42:53 dist-test-slave-cjc6 krb5kdc[13130](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781955773, etypes {rep=17 tkt=17 ses=17}, oryx/127.6.234.252@KRBTEST.COM for oryx/127.6.234.253@KRBTEST.COM
I20260620 11:42:53.309659 13897 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 11:42:53.303608 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.234.253:41509 (local address 127.0.0.1:38352)
0620 11:42:53.303739 (+ 131us) negotiation.cc:107] Waiting for socket to connect
0620 11:42:53.303752 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0620 11:42:53.303808 (+ 56us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 11:42:53.304016 (+ 208us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 11:42:53.304017 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 11:42:53.304139 (+ 122us) client_negotiation.cc:189] Negotiated authn=SASL
0620 11:42:53.304166 (+ 27us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 11:42:53.304167 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 11:42:53.304843 (+ 676us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 11:42:53.304844 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 11:42:53.305371 (+ 527us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 11:42:53.305372 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 11:42:53.305686 (+ 314us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 11:42:53.305687 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 11:42:53.305821 (+ 134us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 11:42:53.306135 (+ 314us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 11:42:53.306139 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0620 11:42:53.307290 (+ 1151us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 11:42:53.308815 (+ 1525us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 11:42:53.308816 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 11:42:53.308816 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0620 11:42:53.308971 (+ 155us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 11:42:53.309174 (+ 203us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 11:42:53.309175 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 11:42:53.309175 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0620 11:42:53.309235 (+ 60us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 11:42:53.309448 (+ 213us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 11:42:53.309448 (+ 0us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 11:42:53.309482 (+ 34us) client_negotiation.cc:769] Sending connection context
0620 11:42:53.309535 (+ 53us) client_negotiation.cc:240] Negotiation successful
0620 11:42:53.309603 (+ 68us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":117}
I20260620 11:42:53.309669 13824 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 11:42:53.303671 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38352 (local address 127.6.234.253:41509)
0620 11:42:53.303711 (+ 40us) server_negotiation.cc:206] Beginning negotiation
0620 11:42:53.303713 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 11:42:53.303793 (+ 80us) server_negotiation.cc:407] Connection header received
0620 11:42:53.303907 (+ 114us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 11:42:53.303908 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 11:42:53.303923 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 11:42:53.303954 (+ 31us) server_negotiation.cc:226] Negotiated authn=SASL
0620 11:42:53.304243 (+ 289us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 11:42:53.304771 (+ 528us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 11:42:53.305443 (+ 672us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 11:42:53.305625 (+ 182us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 11:42:53.305673 (+ 48us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 11:42:53.307365 (+ 1692us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 11:42:53.307366 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 11:42:53.307367 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 11:42:53.307373 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0620 11:42:53.308739 (+ 1366us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 11:42:53.309037 (+ 298us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 11:42:53.309037 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 11:42:53.309038 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 11:42:53.309086 (+ 48us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 11:42:53.309299 (+ 213us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 11:42:53.309300 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 11:42:53.309300 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 11:42:53.309391 (+ 91us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 11:42:53.309440 (+ 49us) server_negotiation.cc:1035] Waiting for connection context
0620 11:42:53.309550 (+ 110us) server_negotiation.cc:299] Negotiation successful
0620 11:42:53.309611 (+ 61us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":18}
I20260620 11:42:53.310055 13895 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.6.234.252" port: 39551 } has no permanent_uuid. Determining permanent_uuid...
Jun 20 11:42:53 dist-test-slave-cjc6 krb5kdc[13130](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781955773, etypes {rep=17 tkt=17 ses=17}, oryx/127.6.234.252@KRBTEST.COM for oryx/127.6.234.252@KRBTEST.COM
I20260620 11:42:53.315860 13897 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 11:42:53.310307 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.234.252:39551 (local address 127.0.0.1:43396)
0620 11:42:53.310356 (+ 49us) negotiation.cc:107] Waiting for socket to connect
0620 11:42:53.310366 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0620 11:42:53.310402 (+ 36us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 11:42:53.310527 (+ 125us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 11:42:53.310528 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 11:42:53.310653 (+ 125us) client_negotiation.cc:189] Negotiated authn=SASL
0620 11:42:53.310676 (+ 23us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 11:42:53.310677 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 11:42:53.311316 (+ 639us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 11:42:53.311317 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 11:42:53.311771 (+ 454us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 11:42:53.311772 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 11:42:53.312149 (+ 377us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 11:42:53.312150 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 11:42:53.312196 (+ 46us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 11:42:53.312526 (+ 330us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 11:42:53.312530 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0620 11:42:53.313585 (+ 1055us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 11:42:53.315109 (+ 1524us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 11:42:53.315110 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 11:42:53.315110 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0620 11:42:53.315257 (+ 147us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 11:42:53.315426 (+ 169us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 11:42:53.315426 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 11:42:53.315426 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0620 11:42:53.315471 (+ 45us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 11:42:53.315636 (+ 165us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 11:42:53.315637 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 11:42:53.315667 (+ 30us) client_negotiation.cc:769] Sending connection context
0620 11:42:53.315715 (+ 48us) client_negotiation.cc:240] Negotiation successful
0620 11:42:53.315789 (+ 74us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":31}
I20260620 11:42:53.315910 13898 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 11:42:53.310376 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43396 (local address 127.6.234.252:39551)
0620 11:42:53.310410 (+ 34us) server_negotiation.cc:206] Beginning negotiation
0620 11:42:53.310411 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0620 11:42:53.310421 (+ 10us) server_negotiation.cc:407] Connection header received
0620 11:42:53.310452 (+ 31us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 11:42:53.310453 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 11:42:53.310461 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 11:42:53.310525 (+ 64us) server_negotiation.cc:226] Negotiated authn=SASL
0620 11:42:53.310782 (+ 257us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 11:42:53.311217 (+ 435us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 11:42:53.311861 (+ 644us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 11:42:53.312051 (+ 190us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 11:42:53.312136 (+ 85us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 11:42:53.313658 (+ 1522us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 11:42:53.313659 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 11:42:53.313659 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 11:42:53.313665 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0620 11:42:53.315043 (+ 1378us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 11:42:53.315324 (+ 281us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 11:42:53.315325 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 11:42:53.315325 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 11:42:53.315371 (+ 46us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 11:42:53.315507 (+ 136us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 11:42:53.315508 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 11:42:53.315508 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 11:42:53.315588 (+ 80us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 11:42:53.315630 (+ 42us) server_negotiation.cc:1035] Waiting for connection context
0620 11:42:53.315729 (+ 99us) server_negotiation.cc:299] Negotiation successful
0620 11:42:53.315829 (+ 100us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":48,"server-negotiator.queue_time_us":20}
I20260620 11:42:53.317282 13895 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P ea509afd379942ea887648a5de383571: Bootstrap starting.
I20260620 11:42:53.318200 13895 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P ea509afd379942ea887648a5de383571: Neither blocks nor log segments found. Creating new log.
I20260620 11:42:53.318488 13895 log.cc:824] T 00000000000000000000000000000000 P ea509afd379942ea887648a5de383571: Log is configured to *not* fsync() on all Append() calls
I20260620 11:42:53.319238 13895 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P ea509afd379942ea887648a5de383571: No bootstrap required, opened a new log
I20260620 11:42:53.320631 13895 raft_consensus.cc:348] T 00000000000000000000000000000000 P ea509afd379942ea887648a5de383571 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cc8526b37e1a4e17aa4785a4a8f15156" member_type: VOTER last_known_addr { host: "127.6.234.254" port: 39029 } } peers { permanent_uuid: "4312e033271940b593bea0332e185bff" member_type: VOTER last_known_addr { host: "127.6.234.253" port: 41509 } } peers { permanent_uuid: "ea509afd379942ea887648a5de383571" member_type: VOTER last_known_addr { host: "127.6.234.252" port: 39551 } }
I20260620 11:42:53.320750 13895 raft_consensus.cc:374] T 00000000000000000000000000000000 P ea509afd379942ea887648a5de383571 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 11:42:53.320773 13895 raft_consensus.cc:729] T 00000000000000000000000000000000 P ea509afd379942ea887648a5de383571 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ea509afd379942ea887648a5de383571, State: Initialized, Role: FOLLOWER
I20260620 11:42:53.320926 13895 consensus_queue.cc:260] T 00000000000000000000000000000000 P ea509afd379942ea887648a5de383571 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cc8526b37e1a4e17aa4785a4a8f15156" member_type: VOTER last_known_addr { host: "127.6.234.254" port: 39029 } } peers { permanent_uuid: "4312e033271940b593bea0332e185bff" member_type: VOTER last_known_addr { host: "127.6.234.253" port: 41509 } } peers { permanent_uuid: "ea509afd379942ea887648a5de383571" member_type: VOTER last_known_addr { host: "127.6.234.252" port: 39551 } }
I20260620 11:42:53.321234 13895 sys_catalog.cc:564] T 00000000000000000000000000000000 P ea509afd379942ea887648a5de383571 [sys.catalog]: configured and running, proceeding with master startup.
I20260620 11:42:53.321201 13900 sys_catalog.cc:455] T 00000000000000000000000000000000 P ea509afd379942ea887648a5de383571 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cc8526b37e1a4e17aa4785a4a8f15156" member_type: VOTER last_known_addr { host: "127.6.234.254" port: 39029 } } peers { permanent_uuid: "4312e033271940b593bea0332e185bff" member_type: VOTER last_known_addr { host: "127.6.234.253" port: 41509 } } peers { permanent_uuid: "ea509afd379942ea887648a5de383571" member_type: VOTER last_known_addr { host: "127.6.234.252" port: 39551 } } }
I20260620 11:42:53.321324 13900 sys_catalog.cc:458] T 00000000000000000000000000000000 P ea509afd379942ea887648a5de383571 [sys.catalog]: This master's current role is: FOLLOWER
W20260620 11:42:53.322818 13911 catalog_manager.cc:1558] T 00000000000000000000000000000000 P ea509afd379942ea887648a5de383571: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260620 11:42:53.322880 13911 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260620 11:42:53.325109 13827 master_runner.cc:428] Detected that this master ea509afd379942ea887648a5de383571 is joining an existing cluster
I20260620 11:42:53.325161 13827 master_runner.cc:432] Initiating AddMaster RPC to add 127.6.234.252:39551
I20260620 11:42:53.346083 13898 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 11:42:53.326603 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43404 (local address 127.6.234.252:39551)
0620 11:42:53.326652 (+ 49us) server_negotiation.cc:206] Beginning negotiation
0620 11:42:53.326653 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0620 11:42:53.326891 (+ 238us) server_negotiation.cc:407] Connection header received
0620 11:42:53.326933 (+ 42us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 11:42:53.326933 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 11:42:53.326942 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 11:42:53.326997 (+ 55us) server_negotiation.cc:226] Negotiated authn=SASL
0620 11:42:53.327336 (+ 339us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 11:42:53.327991 (+ 655us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 11:42:53.329114 (+ 1123us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 11:42:53.329389 (+ 275us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 11:42:53.329427 (+ 38us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 11:42:53.342927 (+ 13500us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 11:42:53.342929 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 11:42:53.342929 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 11:42:53.342973 (+ 44us) server_negotiation.cc:922] calling sasl_server_start()
0620 11:42:53.344793 (+ 1820us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 11:42:53.345227 (+ 434us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 11:42:53.345228 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 11:42:53.345229 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 11:42:53.345289 (+ 60us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 11:42:53.345474 (+ 185us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 11:42:53.345474 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 11:42:53.345474 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 11:42:53.345619 (+ 145us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 11:42:53.345659 (+ 40us) server_negotiation.cc:1035] Waiting for connection context
0620 11:42:53.345884 (+ 225us) server_negotiation.cc:299] Negotiation successful
0620 11:42:53.345969 (+ 85us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":34}
I20260620 11:42:53.346231 13215 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 11:42:53.326217 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51606 (local address 127.6.234.254:39029)
0620 11:42:53.326259 (+ 42us) server_negotiation.cc:206] Beginning negotiation
0620 11:42:53.326261 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 11:42:53.327035 (+ 774us) server_negotiation.cc:407] Connection header received
0620 11:42:53.327126 (+ 91us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 11:42:53.327127 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 11:42:53.327138 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 11:42:53.327167 (+ 29us) server_negotiation.cc:226] Negotiated authn=SASL
0620 11:42:53.327647 (+ 480us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 11:42:53.328403 (+ 756us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 11:42:53.329538 (+ 1135us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 11:42:53.329852 (+ 314us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 11:42:53.329896 (+ 44us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 11:42:53.342965 (+ 13069us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 11:42:53.342967 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 11:42:53.342968 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 11:42:53.342976 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0620 11:42:53.344877 (+ 1901us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 11:42:53.345338 (+ 461us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 11:42:53.345339 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 11:42:53.345339 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 11:42:53.345397 (+ 58us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 11:42:53.345642 (+ 245us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 11:42:53.345643 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 11:42:53.345643 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 11:42:53.345802 (+ 159us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 11:42:53.345845 (+ 43us) server_negotiation.cc:1035] Waiting for connection context
0620 11:42:53.346048 (+ 203us) server_negotiation.cc:299] Negotiation successful
0620 11:42:53.346128 (+ 80us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":20}
I20260620 11:42:53.346498 13917 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 11:42:53.326381 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.234.252:39551 (local address 127.0.0.1:43404)
0620 11:42:53.326823 (+ 442us) negotiation.cc:107] Waiting for socket to connect
0620 11:42:53.326832 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0620 11:42:53.326884 (+ 52us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 11:42:53.327032 (+ 148us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 11:42:53.327033 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 11:42:53.327194 (+ 161us) client_negotiation.cc:189] Negotiated authn=SASL
0620 11:42:53.327264 (+ 70us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 11:42:53.327265 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 11:42:53.328066 (+ 801us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 11:42:53.328066 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 11:42:53.329042 (+ 976us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 11:42:53.329044 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 11:42:53.329518 (+ 474us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 11:42:53.329519 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 11:42:53.329612 (+ 93us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 11:42:53.342218 (+ 12606us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 11:42:53.342225 (+ 7us) client_negotiation.cc:656] calling sasl_client_start()
0620 11:42:53.342802 (+ 577us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 11:42:53.344884 (+ 2082us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 11:42:53.344885 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 11:42:53.344886 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0620 11:42:53.345146 (+ 260us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 11:42:53.345356 (+ 210us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 11:42:53.345357 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 11:42:53.345357 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0620 11:42:53.345410 (+ 53us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 11:42:53.345731 (+ 321us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 11:42:53.345731 (+ 0us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 11:42:53.345776 (+ 45us) client_negotiation.cc:769] Sending connection context
0620 11:42:53.346024 (+ 248us) client_negotiation.cc:240] Negotiation successful
0620 11:42:53.346112 (+ 88us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":426,"mutex_wait_us":73,"thread_start_us":72,"threads_started":1}
I20260620 11:42:53.346721 13919 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 11:42:53.326481 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.234.254:39029 (local address 127.0.0.1:51606)
0620 11:42:53.326979 (+ 498us) negotiation.cc:107] Waiting for socket to connect
0620 11:42:53.326990 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0620 11:42:53.327058 (+ 68us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 11:42:53.327226 (+ 168us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 11:42:53.327227 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 11:42:53.327453 (+ 226us) client_negotiation.cc:189] Negotiated authn=SASL
0620 11:42:53.327519 (+ 66us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 11:42:53.327520 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 11:42:53.328514 (+ 994us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 11:42:53.328514 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 11:42:53.329401 (+ 887us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 11:42:53.329403 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 11:42:53.329985 (+ 582us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 11:42:53.329986 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 11:42:53.330056 (+ 70us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 11:42:53.330424 (+ 368us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 11:42:53.330429 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0620 11:42:53.342839 (+ 12410us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 11:42:53.344975 (+ 2136us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 11:42:53.344976 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 11:42:53.344976 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0620 11:42:53.345251 (+ 275us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 11:42:53.345480 (+ 229us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 11:42:53.345480 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 11:42:53.345481 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0620 11:42:53.345572 (+ 91us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 11:42:53.345893 (+ 321us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 11:42:53.345893 (+ 0us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 11:42:53.345936 (+ 43us) client_negotiation.cc:769] Sending connection context
0620 11:42:53.346397 (+ 461us) client_negotiation.cc:240] Negotiation successful
0620 11:42:53.346512 (+ 115us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":473,"thread_start_us":75,"threads_started":1}
I20260620 11:42:53.347509 13824 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 11:42:53.326563 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38358 (local address 127.6.234.253:41509)
0620 11:42:53.326650 (+ 87us) server_negotiation.cc:206] Beginning negotiation
0620 11:42:53.326652 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 11:42:53.326660 (+ 8us) server_negotiation.cc:407] Connection header received
0620 11:42:53.326681 (+ 21us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 11:42:53.326682 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 11:42:53.326691 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 11:42:53.326754 (+ 63us) server_negotiation.cc:226] Negotiated authn=SASL
0620 11:42:53.327165 (+ 411us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 11:42:53.327952 (+ 787us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 11:42:53.329091 (+ 1139us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 11:42:53.329388 (+ 297us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 11:42:53.329427 (+ 39us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 11:42:53.342943 (+ 13516us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 11:42:53.342945 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 11:42:53.342945 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 11:42:53.342977 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0620 11:42:53.345485 (+ 2508us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 11:42:53.345881 (+ 396us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 11:42:53.345882 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 11:42:53.345882 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 11:42:53.345937 (+ 55us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 11:42:53.346513 (+ 576us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 11:42:53.346515 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 11:42:53.346515 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 11:42:53.346660 (+ 145us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 11:42:53.346705 (+ 45us) server_negotiation.cc:1035] Waiting for connection context
0620 11:42:53.347348 (+ 643us) server_negotiation.cc:299] Negotiation successful
0620 11:42:53.347424 (+ 76us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":65}
I20260620 11:42:53.347513 13918 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 11:42:53.326199 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.234.253:41509 (local address 127.0.0.1:38358)
0620 11:42:53.326521 (+ 322us) negotiation.cc:107] Waiting for socket to connect
0620 11:42:53.326534 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0620 11:42:53.326586 (+ 52us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 11:42:53.326786 (+ 200us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 11:42:53.326787 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 11:42:53.326984 (+ 197us) client_negotiation.cc:189] Negotiated authn=SASL
0620 11:42:53.327071 (+ 87us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 11:42:53.327073 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 11:42:53.328043 (+ 970us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 11:42:53.328044 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 11:42:53.328992 (+ 948us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 11:42:53.328995 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 11:42:53.329516 (+ 521us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 11:42:53.329517 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 11:42:53.329584 (+ 67us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 11:42:53.330105 (+ 521us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 11:42:53.330111 (+ 6us) client_negotiation.cc:656] calling sasl_client_start()
0620 11:42:53.342843 (+ 12732us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 11:42:53.345571 (+ 2728us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 11:42:53.345571 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 11:42:53.345571 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0620 11:42:53.345799 (+ 228us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 11:42:53.346008 (+ 209us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 11:42:53.346009 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 11:42:53.346009 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0620 11:42:53.346423 (+ 414us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 11:42:53.347176 (+ 753us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 11:42:53.347176 (+ 0us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 11:42:53.347221 (+ 45us) client_negotiation.cc:769] Sending connection context
0620 11:42:53.347310 (+ 89us) client_negotiation.cc:240] Negotiation successful
0620 11:42:53.347414 (+ 104us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":295,"mutex_wait_us":36,"thread_start_us":74,"threads_started":1}
Jun 20 11:42:53 dist-test-slave-cjc6 krb5kdc[13130](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781955772, etypes {rep=17 tkt=17 ses=17}, oryx/127.6.234.254@KRBTEST.COM for oryx/127.6.234.252@KRBTEST.COM
I20260620 11:42:53.355033 7083 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task1JRg7N/build/release/bin/kudu with pid 13237
I20260620 11:42:53.355659 13733 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 11:42:53.348986 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.234.252:39551 (local address 127.0.0.1:43410)
0620 11:42:53.349045 (+ 59us) negotiation.cc:107] Waiting for socket to connect
0620 11:42:53.349056 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0620 11:42:53.349098 (+ 42us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 11:42:53.349269 (+ 171us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 11:42:53.349270 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 11:42:53.349422 (+ 152us) client_negotiation.cc:189] Negotiated authn=SASL
0620 11:42:53.349462 (+ 40us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 11:42:53.349463 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 11:42:53.350143 (+ 680us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 11:42:53.350144 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 11:42:53.350666 (+ 522us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 11:42:53.350667 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 11:42:53.351047 (+ 380us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 11:42:53.351047 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 11:42:53.351114 (+ 67us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 11:42:53.351545 (+ 431us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 11:42:53.351550 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0620 11:42:53.353030 (+ 1480us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 11:42:53.354581 (+ 1551us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 11:42:53.354582 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 11:42:53.354582 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0620 11:42:53.354781 (+ 199us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 11:42:53.355057 (+ 276us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 11:42:53.355058 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 11:42:53.355059 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0620 11:42:53.355122 (+ 63us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 11:42:53.355375 (+ 253us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 11:42:53.355376 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 11:42:53.355417 (+ 41us) client_negotiation.cc:769] Sending connection context
0620 11:42:53.355472 (+ 55us) client_negotiation.cc:240] Negotiation successful
0620 11:42:53.355572 (+ 100us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":41}
I20260620 11:42:53.355955 13898 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 11:42:53.349043 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43410 (local address 127.6.234.252:39551)
0620 11:42:53.349081 (+ 38us) server_negotiation.cc:206] Beginning negotiation
0620 11:42:53.349083 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 11:42:53.349092 (+ 9us) server_negotiation.cc:407] Connection header received
0620 11:42:53.349166 (+ 74us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 11:42:53.349167 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 11:42:53.349176 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 11:42:53.349263 (+ 87us) server_negotiation.cc:226] Negotiated authn=SASL
0620 11:42:53.349538 (+ 275us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 11:42:53.350063 (+ 525us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 11:42:53.350765 (+ 702us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 11:42:53.350962 (+ 197us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 11:42:53.351046 (+ 84us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 11:42:53.353125 (+ 2079us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 11:42:53.353127 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 11:42:53.353127 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 11:42:53.353134 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0620 11:42:53.354499 (+ 1365us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 11:42:53.354940 (+ 441us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 11:42:53.354941 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 11:42:53.354941 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 11:42:53.354987 (+ 46us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 11:42:53.355174 (+ 187us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 11:42:53.355176 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 11:42:53.355176 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 11:42:53.355298 (+ 122us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 11:42:53.355352 (+ 54us) server_negotiation.cc:1035] Waiting for connection context
0620 11:42:53.355800 (+ 448us) server_negotiation.cc:299] Negotiation successful
0620 11:42:53.355881 (+ 81us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":19}
I20260620 11:42:53.360029 13175 catalog_manager.cc:6979] Initiating ChangeConfig request to add master 127.6.234.252:39551: tablet_id: "00000000000000000000000000000000"
type: ADD_PEER
server {
permanent_uuid: "ea509afd379942ea887648a5de383571"
member_type: NON_VOTER
last_known_addr {
host: "127.6.234.252"
port: 39551
}
attrs {
promote: true
}
}
dest_uuid: "cc8526b37e1a4e17aa4785a4a8f15156"
cas_config_opid_index: 5
I20260620 11:42:53.360303 13175 consensus_queue.cc:237] T 00000000000000000000000000000000 P cc8526b37e1a4e17aa4785a4a8f15156 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 1.5, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "cc8526b37e1a4e17aa4785a4a8f15156" member_type: VOTER last_known_addr { host: "127.6.234.254" port: 39029 } } peers { permanent_uuid: "4312e033271940b593bea0332e185bff" member_type: NON_VOTER last_known_addr { host: "127.6.234.253" port: 41509 } attrs { promote: true } } peers { permanent_uuid: "ea509afd379942ea887648a5de383571" member_type: NON_VOTER last_known_addr { host: "127.6.234.252" port: 39551 } attrs { promote: true } }
I20260620 11:42:53.360652 13735 raft_consensus.cc:2932] T 00000000000000000000000000000000 P cc8526b37e1a4e17aa4785a4a8f15156 [term 1 LEADER]: Committing config change with OpId 1.6: config changed from index 5 to 6, NON_VOTER ea509afd379942ea887648a5de383571 (127.6.234.252) added. New config: { opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "cc8526b37e1a4e17aa4785a4a8f15156" member_type: VOTER last_known_addr { host: "127.6.234.254" port: 39029 } } peers { permanent_uuid: "4312e033271940b593bea0332e185bff" member_type: NON_VOTER last_known_addr { host: "127.6.234.253" port: 41509 } attrs { promote: true } } peers { permanent_uuid: "ea509afd379942ea887648a5de383571" member_type: NON_VOTER last_known_addr { host: "127.6.234.252" port: 39551 } attrs { promote: true } } }
I20260620 11:42:53.361567 13735 catalog_manager.cc:6983] Successfully completed master ChangeConfig request to add master 127.6.234.252:39551
I20260620 11:42:53.361657 13735 sys_catalog.cc:455] T 00000000000000000000000000000000 P cc8526b37e1a4e17aa4785a4a8f15156 [sys.catalog]: SysCatalogTable state changed. Reason: Config change replication complete. Latest consensus state: current_term: 1 leader_uuid: "cc8526b37e1a4e17aa4785a4a8f15156" committed_config { opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "cc8526b37e1a4e17aa4785a4a8f15156" member_type: VOTER last_known_addr { host: "127.6.234.254" port: 39029 } } peers { permanent_uuid: "4312e033271940b593bea0332e185bff" member_type: NON_VOTER last_known_addr { host: "127.6.234.253" port: 41509 } attrs { promote: true } } peers { permanent_uuid: "ea509afd379942ea887648a5de383571" member_type: NON_VOTER last_known_addr { host: "127.6.234.252" port: 39551 } attrs { promote: true } } }
I20260620 11:42:53.361765 13735 sys_catalog.cc:458] T 00000000000000000000000000000000 P cc8526b37e1a4e17aa4785a4a8f15156 [sys.catalog]: This master's current role is: LEADER
I20260620 11:42:53.362026 13869 raft_consensus.cc:3037] T 00000000000000000000000000000000 P ea509afd379942ea887648a5de383571 [term 0 FOLLOWER]: Advancing to term 1
I20260620 11:42:53.362692 7083 external_mini_cluster.cc:1326] Running /tmp/dist-test-task1JRg7N/build/release/bin/kudu
/tmp/dist-test-task1JRg7N/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task1JRg7N/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781955746769051-7083-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-task1JRg7N/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781955746769051-7083-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task1JRg7N/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781955746769051-7083-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-task1JRg7N/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781955746769051-7083-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.6.234.193:38171
--local_ip_for_outbound_sockets=127.6.234.193
--tserver_master_addrs=127.6.234.254:39029
--webserver_port=37865
--webserver_interface=127.6.234.193
--keytab_file=/tmp/dist-test-task1JRg7N/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781955746769051-7083-0/krb5kdc/oryx_127.6.234.193.keytab
--principal=oryx/127.6.234.193
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.6.234.212:38681
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_master_addrs=127.6.234.254:39029,127.6.234.253:41509
--tserver_master_addrs=127.6.234.254:39029,127.6.234.253:41509,127.6.234.252:39551 with env {KRB5CCNAME=/tmp/dist-test-task1JRg7N/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781955746769051-7083-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task1JRg7N/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781955746769051-7083-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task1JRg7N/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781955746769051-7083-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260620 11:42:53.363018 13869 raft_consensus.cc:1261] T 00000000000000000000000000000000 P ea509afd379942ea887648a5de383571 [term 1 FOLLOWER]: Refusing update from remote peer cc8526b37e1a4e17aa4785a4a8f15156: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20260620 11:42:53.363394 13734 consensus_queue.cc:1035] T 00000000000000000000000000000000 P cc8526b37e1a4e17aa4785a4a8f15156 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ea509afd379942ea887648a5de383571" member_type: NON_VOTER last_known_addr { host: "127.6.234.252" port: 39551 } attrs { promote: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 11:42:53.364116 13827 master.cc:537] Master@127.6.234.252:39551 shutting down...
I20260620 11:42:53.364784 13869 raft_consensus.cc:858] T 00000000000000000000000000000000 P ea509afd379942ea887648a5de383571 [term 1 NON_PARTICIPANT]: Illegal state: RaftConfig change currently pending. Only one is allowed at a time.
Committed config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cc8526b37e1a4e17aa4785a4a8f15156" member_type: VOTER last_known_addr { host: "127.6.234.254" port: 39029 } } peers { permanent_uuid: "4312e033271940b593bea0332e185bff" member_type: VOTER last_known_addr { host: "127.6.234.253" port: 41509 } } peers { permanent_uuid: "ea509afd379942ea887648a5de383571" member_type: VOTER last_known_addr { host: "127.6.234.252" port: 39551 } }.
Pending config: opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "cc8526b37e1a4e17aa4785a4a8f15156" member_type: VOTER last_known_addr { host: "127.6.234.254" port: 39029 } } peers { permanent_uuid: "4312e033271940b593bea0332e185bff" member_type: NON_VOTER last_known_addr { host: "127.6.234.253" port: 41509 } attrs { promote: true } }:
New config: opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "cc8526b37e1a4e17aa4785a4a8f15156" member_type: VOTER last_known_addr { host: "127.6.234.254" port: 39029 } } peers { permanent_uuid: "4312e033271940b593bea0332e185bff" member_type: NON_VOTER last_known_addr { host: "127.6.234.253" port: 41509 } attrs { promote: true } } peers { permanent_uuid: "ea509afd379942ea887648a5de383571" member_type: NON_VOTER last_known_addr { host: "127.6.234.252" port: 39551 } attrs { promote: true } }
W20260620 11:42:53.364876 13869 raft_consensus.cc:1565] T 00000000000000000000000000000000 P ea509afd379942ea887648a5de383571 [term 1 NON_PARTICIPANT]: Could not prepare op 'term: 1 index: 6' and following 0 ops. Status for this op: Illegal state: RaftConfig change currently pending. Only one is allowed at a time.
Committed config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cc8526b37e1a4e17aa4785a4a8f15156" member_type: VOTER last_known_addr { host: "127.6.234.254" port: 39029 } } peers { permanent_uuid: "4312e033271940b593bea0332e185bff" member_type: VOTER last_known_addr { host: "127.6.234.253" port: 41509 } } peers { permanent_uuid: "ea509afd379942ea887648a5de383571" member_type: VOTER last_known_addr { host: "127.6.234.252" port: 39551 } }.
Pending config: opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "cc8526b37e1a4e17aa4785a4a8f15156" member_type: VOTER last_known_addr { host: "127.6.234.254" port: 39029 } } peers { permanent_uuid: "4312e033271940b593bea0332e185bff" member_type: NON_VOTER last_known_addr { host: "127.6.234.253" port: 41509 } attrs { promote: true } }:
New config: opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "cc8526b37e1a4e17aa4785a4a8f15156" member_type: VOTER last_known_addr { host: "127.6.234.254" port: 39029 } } peers { permanent_uuid: "4312e033271940b593bea0332e185bff" member_type: NON_VOTER last_known_addr { host: "127.6.234.253" port: 41509 } attrs { promote: true } } peers { permanent_uuid: "ea509afd379942ea887648a5de383571" member_type: NON_VOTER last_known_addr { host: "127.6.234.252" port: 39551 } attrs { promote: true } }
I20260620 11:42:53.365327 13869 raft_consensus.cc:2932] T 00000000000000000000000000000000 P ea509afd379942ea887648a5de383571 [term 1 NON_PARTICIPANT]: Committing config change with OpId 1.5: config changed from index -1 to 5, NON_VOTER 4312e033271940b593bea0332e185bff (127.6.234.253) added. New config: { opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "cc8526b37e1a4e17aa4785a4a8f15156" member_type: VOTER last_known_addr { host: "127.6.234.254" port: 39029 } } peers { permanent_uuid: "4312e033271940b593bea0332e185bff" member_type: NON_VOTER last_known_addr { host: "127.6.234.253" port: 41509 } attrs { promote: true } } }
I20260620 11:42:53.366739 13923 sys_catalog.cc:455] T 00000000000000000000000000000000 P ea509afd379942ea887648a5de383571 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "cc8526b37e1a4e17aa4785a4a8f15156" committed_config { opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "cc8526b37e1a4e17aa4785a4a8f15156" member_type: VOTER last_known_addr { host: "127.6.234.254" port: 39029 } } peers { permanent_uuid: "4312e033271940b593bea0332e185bff" member_type: NON_VOTER last_known_addr { host: "127.6.234.253" port: 41509 } attrs { promote: true } } }
I20260620 11:42:53.366838 13923 sys_catalog.cc:458] T 00000000000000000000000000000000 P ea509afd379942ea887648a5de383571 [sys.catalog]: This master's current role is: NON_PARTICIPANT
I20260620 11:42:53.366878 13923 sys_catalog.cc:455] T 00000000000000000000000000000000 P ea509afd379942ea887648a5de383571 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "cc8526b37e1a4e17aa4785a4a8f15156" committed_config { opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "cc8526b37e1a4e17aa4785a4a8f15156" member_type: VOTER last_known_addr { host: "127.6.234.254" port: 39029 } } peers { permanent_uuid: "4312e033271940b593bea0332e185bff" member_type: NON_VOTER last_known_addr { host: "127.6.234.253" port: 41509 } attrs { promote: true } } }
I20260620 11:42:53.366930 13923 sys_catalog.cc:458] T 00000000000000000000000000000000 P ea509afd379942ea887648a5de383571 [sys.catalog]: This master's current role is: NON_PARTICIPANT
I20260620 11:42:53.367297 13900 sys_catalog.cc:455] T 00000000000000000000000000000000 P ea509afd379942ea887648a5de383571 [sys.catalog]: SysCatalogTable state changed. Reason: New leader cc8526b37e1a4e17aa4785a4a8f15156. Latest consensus state: current_term: 1 leader_uuid: "cc8526b37e1a4e17aa4785a4a8f15156" committed_config { opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "cc8526b37e1a4e17aa4785a4a8f15156" member_type: VOTER last_known_addr { host: "127.6.234.254" port: 39029 } } peers { permanent_uuid: "4312e033271940b593bea0332e185bff" member_type: NON_VOTER last_known_addr { host: "127.6.234.253" port: 41509 } attrs { promote: true } } }
I20260620 11:42:53.367369 13900 sys_catalog.cc:458] T 00000000000000000000000000000000 P ea509afd379942ea887648a5de383571 [sys.catalog]: This master's current role is: NON_PARTICIPANT
I20260620 11:42:53.367743 13734 sys_catalog.cc:455] T 00000000000000000000000000000000 P cc8526b37e1a4e17aa4785a4a8f15156 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "cc8526b37e1a4e17aa4785a4a8f15156" committed_config { opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "cc8526b37e1a4e17aa4785a4a8f15156" member_type: VOTER last_known_addr { host: "127.6.234.254" port: 39029 } } peers { permanent_uuid: "4312e033271940b593bea0332e185bff" member_type: NON_VOTER last_known_addr { host: "127.6.234.253" port: 41509 } attrs { promote: true } } peers { permanent_uuid: "ea509afd379942ea887648a5de383571" member_type: NON_VOTER last_known_addr { host: "127.6.234.252" port: 39551 } attrs { promote: true } } }
I20260620 11:42:53.367846 13734 sys_catalog.cc:458] T 00000000000000000000000000000000 P cc8526b37e1a4e17aa4785a4a8f15156 [sys.catalog]: This master's current role is: LEADER
W20260620 11:42:53.368500 13161 consensus_peers.cc:487] T 00000000000000000000000000000000 P cc8526b37e1a4e17aa4785a4a8f15156 -> Peer ea509afd379942ea887648a5de383571 (127.6.234.252:39551): Couldn't send request to peer ea509afd379942ea887648a5de383571. Status: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on Master. This is attempt 1: this message will repeat every 5th retry.
I20260620 11:42:53.368712 13827 raft_consensus.cc:2229] T 00000000000000000000000000000000 P ea509afd379942ea887648a5de383571 [term 1 NON_PARTICIPANT]: Raft consensus shutting down.
I20260620 11:42:53.368796 13827 raft_consensus.cc:2258] T 00000000000000000000000000000000 P ea509afd379942ea887648a5de383571 [term 1 NON_PARTICIPANT]: Raft consensus is shut down!
I20260620 11:42:53.368827 13827 tablet_replica.cc:330] T 00000000000000000000000000000000 P ea509afd379942ea887648a5de383571: stopping tablet replica
I20260620 11:42:53.370440 13733 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 11:42:53.360856 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.234.253:41509 (local address 127.0.0.1:38360)
0620 11:42:53.361070 (+ 214us) negotiation.cc:107] Waiting for socket to connect
0620 11:42:53.361082 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0620 11:42:53.361118 (+ 36us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 11:42:53.362666 (+ 1548us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 11:42:53.362667 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 11:42:53.362669 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 11:42:53.362709 (+ 40us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 11:42:53.362711 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 11:42:53.363962 (+ 1251us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 11:42:53.363963 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 11:42:53.369404 (+ 5441us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 11:42:53.369412 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 11:42:53.370206 (+ 794us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 11:42:53.370207 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 11:42:53.370264 (+ 57us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 11:42:53.370300 (+ 36us) client_negotiation.cc:769] Sending connection context
0620 11:42:53.370351 (+ 51us) client_negotiation.cc:240] Negotiation successful
0620 11:42:53.370386 (+ 35us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":197}
I20260620 11:42:53.370724 13824 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 11:42:53.362424 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38360 (local address 127.6.234.253:41509)
0620 11:42:53.362472 (+ 48us) server_negotiation.cc:206] Beginning negotiation
0620 11:42:53.362474 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 11:42:53.362483 (+ 9us) server_negotiation.cc:407] Connection header received
0620 11:42:53.362509 (+ 26us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 11:42:53.362510 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 11:42:53.362532 (+ 22us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 11:42:53.362599 (+ 67us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 11:42:53.363059 (+ 460us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 11:42:53.363777 (+ 718us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 11:42:53.369497 (+ 5720us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 11:42:53.370130 (+ 633us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 11:42:53.370210 (+ 80us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 11:42:53.370290 (+ 80us) server_negotiation.cc:1035] Waiting for connection context
0620 11:42:53.370612 (+ 322us) server_negotiation.cc:299] Negotiation successful
0620 11:42:53.370655 (+ 43us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":28}
I20260620 11:42:53.371001 13783 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 4312e033271940b593bea0332e185bff [term 1 LEARNER]: Refusing update from remote peer cc8526b37e1a4e17aa4785a4a8f15156: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20260620 11:42:53.371201 13735 consensus_queue.cc:1035] T 00000000000000000000000000000000 P cc8526b37e1a4e17aa4785a4a8f15156 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4312e033271940b593bea0332e185bff" member_type: NON_VOTER last_known_addr { host: "127.6.234.253" port: 41509 } attrs { promote: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20260620 11:42:53.371835 13783 raft_consensus.cc:2932] T 00000000000000000000000000000000 P 4312e033271940b593bea0332e185bff [term 1 LEARNER]: Committing config change with OpId 1.6: config changed from index 5 to 6, NON_VOTER ea509afd379942ea887648a5de383571 (127.6.234.252) added. New config: { opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "cc8526b37e1a4e17aa4785a4a8f15156" member_type: VOTER last_known_addr { host: "127.6.234.254" port: 39029 } } peers { permanent_uuid: "4312e033271940b593bea0332e185bff" member_type: NON_VOTER last_known_addr { host: "127.6.234.253" port: 41509 } attrs { promote: true } } peers { permanent_uuid: "ea509afd379942ea887648a5de383571" member_type: NON_VOTER last_known_addr { host: "127.6.234.252" port: 39551 } attrs { promote: true } } }
I20260620 11:42:53.373010 13735 raft_consensus.cc:1050] T 00000000000000000000000000000000 P cc8526b37e1a4e17aa4785a4a8f15156: attempting to promote NON_VOTER 4312e033271940b593bea0332e185bff to VOTER
I20260620 11:42:53.373204 13735 consensus_queue.cc:237] T 00000000000000000000000000000000 P cc8526b37e1a4e17aa4785a4a8f15156 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 1.6, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "cc8526b37e1a4e17aa4785a4a8f15156" member_type: VOTER last_known_addr { host: "127.6.234.254" port: 39029 } } peers { permanent_uuid: "4312e033271940b593bea0332e185bff" member_type: VOTER last_known_addr { host: "127.6.234.253" port: 41509 } attrs { promote: false } } peers { permanent_uuid: "ea509afd379942ea887648a5de383571" member_type: NON_VOTER last_known_addr { host: "127.6.234.252" port: 39551 } attrs { promote: true } }
I20260620 11:42:53.373237 13812 sys_catalog.cc:455] T 00000000000000000000000000000000 P 4312e033271940b593bea0332e185bff [sys.catalog]: SysCatalogTable state changed. Reason: New leader cc8526b37e1a4e17aa4785a4a8f15156. Latest consensus state: current_term: 1 leader_uuid: "cc8526b37e1a4e17aa4785a4a8f15156" committed_config { opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "cc8526b37e1a4e17aa4785a4a8f15156" member_type: VOTER last_known_addr { host: "127.6.234.254" port: 39029 } } peers { permanent_uuid: "4312e033271940b593bea0332e185bff" member_type: NON_VOTER last_known_addr { host: "127.6.234.253" port: 41509 } attrs { promote: true } } peers { permanent_uuid: "ea509afd379942ea887648a5de383571" member_type: NON_VOTER last_known_addr { host: "127.6.234.252" port: 39551 } attrs { promote: true } } }
I20260620 11:42:53.373322 13812 sys_catalog.cc:458] T 00000000000000000000000000000000 P 4312e033271940b593bea0332e185bff [sys.catalog]: This master's current role is: LEARNER
I20260620 11:42:53.373360 13812 sys_catalog.cc:455] T 00000000000000000000000000000000 P 4312e033271940b593bea0332e185bff [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "cc8526b37e1a4e17aa4785a4a8f15156" committed_config { opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "cc8526b37e1a4e17aa4785a4a8f15156" member_type: VOTER last_known_addr { host: "127.6.234.254" port: 39029 } } peers { permanent_uuid: "4312e033271940b593bea0332e185bff" member_type: NON_VOTER last_known_addr { host: "127.6.234.253" port: 41509 } attrs { promote: true } } peers { permanent_uuid: "ea509afd379942ea887648a5de383571" member_type: NON_VOTER last_known_addr { host: "127.6.234.252" port: 39551 } attrs { promote: true } } }
I20260620 11:42:53.373417 13812 sys_catalog.cc:458] T 00000000000000000000000000000000 P 4312e033271940b593bea0332e185bff [sys.catalog]: This master's current role is: LEARNER
I20260620 11:42:53.373986 13783 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 4312e033271940b593bea0332e185bff [term 1 LEARNER]: Refusing update from remote peer cc8526b37e1a4e17aa4785a4a8f15156: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
I20260620 11:42:53.374234 13734 consensus_queue.cc:1035] T 00000000000000000000000000000000 P cc8526b37e1a4e17aa4785a4a8f15156 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4312e033271940b593bea0332e185bff" member_type: VOTER last_known_addr { host: "127.6.234.253" port: 41509 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
W20260620 11:42:53.374310 13161 consensus_peers.cc:487] T 00000000000000000000000000000000 P cc8526b37e1a4e17aa4785a4a8f15156 -> Peer ea509afd379942ea887648a5de383571 (127.6.234.252:39551): Couldn't send request to peer ea509afd379942ea887648a5de383571. Status: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on Master. This is attempt 1: this message will repeat every 5th retry.
I20260620 11:42:53.374760 13735 raft_consensus.cc:2932] T 00000000000000000000000000000000 P cc8526b37e1a4e17aa4785a4a8f15156 [term 1 LEADER]: Committing config change with OpId 1.7: config changed from index 6 to 7, 4312e033271940b593bea0332e185bff (127.6.234.253) changed from NON_VOTER to VOTER. New config: { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "cc8526b37e1a4e17aa4785a4a8f15156" member_type: VOTER last_known_addr { host: "127.6.234.254" port: 39029 } } peers { permanent_uuid: "4312e033271940b593bea0332e185bff" member_type: VOTER last_known_addr { host: "127.6.234.253" port: 41509 } attrs { promote: false } } peers { permanent_uuid: "ea509afd379942ea887648a5de383571" member_type: NON_VOTER last_known_addr { host: "127.6.234.252" port: 39551 } attrs { promote: true } } }
I20260620 11:42:53.375597 13735 sys_catalog.cc:455] T 00000000000000000000000000000000 P cc8526b37e1a4e17aa4785a4a8f15156 [sys.catalog]: SysCatalogTable state changed. Reason: Config change replication complete. Latest consensus state: current_term: 1 leader_uuid: "cc8526b37e1a4e17aa4785a4a8f15156" committed_config { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "cc8526b37e1a4e17aa4785a4a8f15156" member_type: VOTER last_known_addr { host: "127.6.234.254" port: 39029 } } peers { permanent_uuid: "4312e033271940b593bea0332e185bff" member_type: VOTER last_known_addr { host: "127.6.234.253" port: 41509 } attrs { promote: false } } peers { permanent_uuid: "ea509afd379942ea887648a5de383571" member_type: NON_VOTER last_known_addr { host: "127.6.234.252" port: 39551 } attrs { promote: true } } }
I20260620 11:42:53.375672 13735 sys_catalog.cc:458] T 00000000000000000000000000000000 P cc8526b37e1a4e17aa4785a4a8f15156 [sys.catalog]: This master's current role is: LEADER
I20260620 11:42:53.375730 13735 sys_catalog.cc:455] T 00000000000000000000000000000000 P cc8526b37e1a4e17aa4785a4a8f15156 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "cc8526b37e1a4e17aa4785a4a8f15156" committed_config { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "cc8526b37e1a4e17aa4785a4a8f15156" member_type: VOTER last_known_addr { host: "127.6.234.254" port: 39029 } } peers { permanent_uuid: "4312e033271940b593bea0332e185bff" member_type: VOTER last_known_addr { host: "127.6.234.253" port: 41509 } attrs { promote: false } } peers { permanent_uuid: "ea509afd379942ea887648a5de383571" member_type: NON_VOTER last_known_addr { host: "127.6.234.252" port: 39551 } attrs { promote: true } } }
I20260620 11:42:53.375785 13735 sys_catalog.cc:458] T 00000000000000000000000000000000 P cc8526b37e1a4e17aa4785a4a8f15156 [sys.catalog]: This master's current role is: LEADER
I20260620 11:42:53.376376 13783 raft_consensus.cc:2932] T 00000000000000000000000000000000 P 4312e033271940b593bea0332e185bff [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7, 4312e033271940b593bea0332e185bff (127.6.234.253) changed from NON_VOTER to VOTER. New config: { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "cc8526b37e1a4e17aa4785a4a8f15156" member_type: VOTER last_known_addr { host: "127.6.234.254" port: 39029 } } peers { permanent_uuid: "4312e033271940b593bea0332e185bff" member_type: VOTER last_known_addr { host: "127.6.234.253" port: 41509 } attrs { promote: false } } peers { permanent_uuid: "ea509afd379942ea887648a5de383571" member_type: NON_VOTER last_known_addr { host: "127.6.234.252" port: 39551 } attrs { promote: true } } }
I20260620 11:42:53.377879 13929 sys_catalog.cc:455] T 00000000000000000000000000000000 P 4312e033271940b593bea0332e185bff [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "cc8526b37e1a4e17aa4785a4a8f15156" committed_config { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "cc8526b37e1a4e17aa4785a4a8f15156" member_type: VOTER last_known_addr { host: "127.6.234.254" port: 39029 } } peers { permanent_uuid: "4312e033271940b593bea0332e185bff" member_type: VOTER last_known_addr { host: "127.6.234.253" port: 41509 } attrs { promote: false } } peers { permanent_uuid: "ea509afd379942ea887648a5de383571" member_type: NON_VOTER last_known_addr { host: "127.6.234.252" port: 39551 } attrs { promote: true } } }
I20260620 11:42:53.378404 13929 sys_catalog.cc:458] T 00000000000000000000000000000000 P 4312e033271940b593bea0332e185bff [sys.catalog]: This master's current role is: FOLLOWER
I20260620 11:42:53.389577 13827 master.cc:552] Master@127.6.234.252:39551 shutdown complete.
I20260620 11:42:53.389642 13827 master_runner.cc:305] Clearing existing system tablet
I20260620 11:42:53.389853 13827 fs_manager.cc:359] Metadata directory not provided
I20260620 11:42:53.389917 13827 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 11:42:53.390518 13827 fs_manager.cc:709] Time spent opening directory manager: real 0.000s user 0.000s sys 0.001s
I20260620 11:42:53.391031 13932 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 11:42:53.391148 13827 fs_report.cc:383] FS layout report
--------------------
wal directory:
metadata directory:
1 data directories: /tmp/dist-test-task1JRg7N/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781955746769051-7083-0/minicluster-data/master-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 11:42:53.391196 13827 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.000s
I20260620 11:42:53.391224 13827 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task1JRg7N/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781955746769051-7083-0/minicluster-data/master-2/data,/tmp/dist-test-task1JRg7N/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781955746769051-7083-0/minicluster-data/master-2/wal
uuid: "ea509afd379942ea887648a5de383571"
format_stamp: "Formatted at 2026-06-20 11:42:53 on dist-test-slave-cjc6"
server_key: "b789541cb9267110241b5dc5b2ff9bd5"
server_key_iv: "8841afa5c047fc6f3706f7d306db14e8"
server_key_version: "encryptionkey@0"
I20260620 11:42:53.391805 13827 ts_tablet_manager.cc:1905] T 00000000000000000000000000000000 P ea509afd379942ea887648a5de383571: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 11:42:53.393092 13827 ts_tablet_manager.cc:1918] T 00000000000000000000000000000000 P ea509afd379942ea887648a5de383571: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId unknown
I20260620 11:42:53.393174 13827 log.cc:1196] T 00000000000000000000000000000000 P ea509afd379942ea887648a5de383571: Deleting WAL directory at /tmp/dist-test-task1JRg7N/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781955746769051-7083-0/minicluster-data/master-2/wal/wals/00000000000000000000000000000000
I20260620 11:42:53.393390 13827 ts_tablet_manager.cc:1939] T 00000000000000000000000000000000 P ea509afd379942ea887648a5de383571: Deleting consensus metadata
I20260620 11:42:53.393525 13827 master_runner.cc:315] Copying system tablet from 127.6.234.254:39029
I20260620 11:42:53.394227 13827 tablet_copy_client.cc:316] T 00000000000000000000000000000000 P ea509afd379942ea887648a5de383571: tablet copy: Beginning tablet copy session from remote peer at address 127.6.234.254:39029
I20260620 11:42:53.400580 13215 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 11:42:53.394561 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51618 (local address 127.6.234.254:39029)
0620 11:42:53.394605 (+ 44us) server_negotiation.cc:206] Beginning negotiation
0620 11:42:53.394607 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 11:42:53.394735 (+ 128us) server_negotiation.cc:407] Connection header received
0620 11:42:53.394824 (+ 89us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 11:42:53.394825 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 11:42:53.394836 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 11:42:53.394872 (+ 36us) server_negotiation.cc:226] Negotiated authn=SASL
0620 11:42:53.395270 (+ 398us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 11:42:53.395831 (+ 561us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 11:42:53.396632 (+ 801us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 11:42:53.396842 (+ 210us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 11:42:53.396917 (+ 75us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 11:42:53.397998 (+ 1081us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 11:42:53.398000 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 11:42:53.398001 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 11:42:53.398008 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0620 11:42:53.399442 (+ 1434us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 11:42:53.399796 (+ 354us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 11:42:53.399797 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 11:42:53.399798 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 11:42:53.399851 (+ 53us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 11:42:53.400057 (+ 206us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 11:42:53.400058 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 11:42:53.400058 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 11:42:53.400180 (+ 122us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 11:42:53.400301 (+ 121us) server_negotiation.cc:1035] Waiting for connection context
0620 11:42:53.400420 (+ 119us) server_negotiation.cc:299] Negotiation successful
0620 11:42:53.400493 (+ 73us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":22}
I20260620 11:42:53.400890 13938 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 11:42:53.394525 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.234.254:39029 (local address 127.0.0.1:51618)
0620 11:42:53.394639 (+ 114us) negotiation.cc:107] Waiting for socket to connect
0620 11:42:53.394675 (+ 36us) client_negotiation.cc:174] Beginning negotiation
0620 11:42:53.394736 (+ 61us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 11:42:53.394925 (+ 189us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 11:42:53.394925 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 11:42:53.395141 (+ 216us) client_negotiation.cc:189] Negotiated authn=SASL
0620 11:42:53.395195 (+ 54us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 11:42:53.395197 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 11:42:53.395926 (+ 729us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 11:42:53.395927 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 11:42:53.396536 (+ 609us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 11:42:53.396538 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 11:42:53.396914 (+ 376us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 11:42:53.396915 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 11:42:53.397040 (+ 125us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 11:42:53.397508 (+ 468us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 11:42:53.397513 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0620 11:42:53.397898 (+ 385us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 11:42:53.399544 (+ 1646us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 11:42:53.399545 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 11:42:53.399546 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0620 11:42:53.399720 (+ 174us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 11:42:53.399925 (+ 205us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 11:42:53.399926 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 11:42:53.399927 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0620 11:42:53.399985 (+ 58us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 11:42:53.400277 (+ 292us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 11:42:53.400278 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 11:42:53.400324 (+ 46us) client_negotiation.cc:769] Sending connection context
0620 11:42:53.400396 (+ 72us) client_negotiation.cc:240] Negotiation successful
0620 11:42:53.400494 (+ 98us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":97,"thread_start_us":61,"threads_started":1}
I20260620 11:42:53.401111 13195 tablet_copy_service.cc:140] P cc8526b37e1a4e17aa4785a4a8f15156: Received BeginTabletCopySession request for tablet 00000000000000000000000000000000 from peer ea509afd379942ea887648a5de383571 ({username='oryx', principal='oryx/127.6.234.252@KRBTEST.COM'} at 127.0.0.1:51618)
I20260620 11:42:53.401160 13195 tablet_copy_service.cc:161] P cc8526b37e1a4e17aa4785a4a8f15156: Beginning new tablet copy session on tablet 00000000000000000000000000000000 from peer ea509afd379942ea887648a5de383571 at {username='oryx', principal='oryx/127.6.234.252@KRBTEST.COM'} at 127.0.0.1:51618: session id = ea509afd379942ea887648a5de383571-00000000000000000000000000000000
I20260620 11:42:53.401620 13195 tablet_copy_source_session.cc:205] T 00000000000000000000000000000000 P cc8526b37e1a4e17aa4785a4a8f15156: Tablet Copy: opened 0 blocks and 1 log segments
I20260620 11:42:53.402112 13827 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total
I20260620 11:42:53.402943 13827 tablet_copy_client.cc:791] T 00000000000000000000000000000000 P ea509afd379942ea887648a5de383571: tablet copy: Starting download of 0 data blocks...
I20260620 11:42:53.403091 13827 tablet_copy_client.cc:655] T 00000000000000000000000000000000 P ea509afd379942ea887648a5de383571: tablet copy: Starting download of 1 WAL segments...
I20260620 11:42:53.403723 13827 tablet_copy_client.cc:531] T 00000000000000000000000000000000 P ea509afd379942ea887648a5de383571: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20260620 11:42:53.404402 13195 tablet_copy_service.cc:342] P cc8526b37e1a4e17aa4785a4a8f15156: Request end of tablet copy session ea509afd379942ea887648a5de383571-00000000000000000000000000000000 received from {username='oryx', principal='oryx/127.6.234.252@KRBTEST.COM'} at 127.0.0.1:51618
I20260620 11:42:53.404477 13195 tablet_copy_service.cc:433] P cc8526b37e1a4e17aa4785a4a8f15156: ending tablet copy session ea509afd379942ea887648a5de383571-00000000000000000000000000000000 on tablet 00000000000000000000000000000000 with peer ea509afd379942ea887648a5de383571
I20260620 11:42:53.406989 13827 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 11:42:53.408351 13939 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 11:42:53.408525 13942 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 11:42:53.408690 13827 server_base.cc:1034] running on GCE node
W20260620 11:42:53.409886 13940 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 11:42:53.410017 13827 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 11:42:53.410153 13827 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 11:42:53.411285 13827 hybrid_clock.cc:648] HybridClock initialized: now 1781955773411266 us; error 38 us; skew 500 ppm
Jun 20 11:42:53 dist-test-slave-cjc6 krb5kdc[13130](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781955773, etypes {rep=17 tkt=17 ses=17}, oryx/127.6.234.252@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 11:42:53.412590 13827 init.cc:377] Logged in from keytab as oryx/127.6.234.252@KRBTEST.COM (short username oryx)
I20260620 11:42:53.412883 13827 webserver.cc:460] Webserver started at http://127.6.234.252:44023/ using document root <none> and password file <none>
I20260620 11:42:53.412998 13827 fs_manager.cc:359] Metadata directory not provided
I20260620 11:42:53.413033 13827 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 11:42:53.413618 13827 fs_manager.cc:709] Time spent opening directory manager: real 0.000s user 0.001s sys 0.000s
I20260620 11:42:53.414155 13949 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 11:42:53.414305 13827 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.000s
I20260620 11:42:53.414350 13827 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task1JRg7N/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781955746769051-7083-0/minicluster-data/master-2/data,/tmp/dist-test-task1JRg7N/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781955746769051-7083-0/minicluster-data/master-2/wal
uuid: "ea509afd379942ea887648a5de383571"
format_stamp: "Formatted at 2026-06-20 11:42:53 on dist-test-slave-cjc6"
server_key: "b789541cb9267110241b5dc5b2ff9bd5"
server_key_iv: "8841afa5c047fc6f3706f7d306db14e8"
server_key_version: "encryptionkey@0"
I20260620 11:42:53.414409 13827 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task1JRg7N/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781955746769051-7083-0/minicluster-data/master-2/wal
metadata directory: /tmp/dist-test-task1JRg7N/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781955746769051-7083-0/minicluster-data/master-2/wal
1 data directories: /tmp/dist-test-task1JRg7N/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781955746769051-7083-0/minicluster-data/master-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 11:42:53.420501 13827 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 11:42:53.422149 13827 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 11:42:53.425271 13827 rpc_server.cc:307] RPC server started. Bound to: 127.6.234.252:39551
I20260620 11:42:53.425527 14001 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.6.234.252:39551 every 8 connection(s)
I20260620 11:42:53.425642 13827 server_base.cc:1166] Dumped server information to /tmp/dist-test-task1JRg7N/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781955746769051-7083-0/minicluster-data/master-2/data/info.pb
I20260620 11:42:53.426461 14002 sys_catalog.cc:263] Verifying existing consensus state
I20260620 11:42:53.426877 14002 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P ea509afd379942ea887648a5de383571: Bootstrap starting.
I20260620 11:42:53.428910 14002 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P ea509afd379942ea887648a5de383571: Bootstrap replayed 1/1 log segments. Stats: ops{read=7 overwritten=0 applied=7 ignored=0} inserts{seen=3 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260620 11:42:53.429131 14002 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P ea509afd379942ea887648a5de383571: Bootstrap complete.
I20260620 11:42:53.429394 14002 raft_consensus.cc:348] T 00000000000000000000000000000000 P ea509afd379942ea887648a5de383571 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "cc8526b37e1a4e17aa4785a4a8f15156" member_type: VOTER last_known_addr { host: "127.6.234.254" port: 39029 } } peers { permanent_uuid: "4312e033271940b593bea0332e185bff" member_type: VOTER last_known_addr { host: "127.6.234.253" port: 41509 } attrs { promote: false } } peers { permanent_uuid: "ea509afd379942ea887648a5de383571" member_type: NON_VOTER last_known_addr { host: "127.6.234.252" port: 39551 } attrs { promote: true } }
I20260620 11:42:53.429483 14002 raft_consensus.cc:729] T 00000000000000000000000000000000 P ea509afd379942ea887648a5de383571 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: ea509afd379942ea887648a5de383571, State: Initialized, Role: LEARNER
I20260620 11:42:53.429529 14002 consensus_queue.cc:260] T 00000000000000000000000000000000 P ea509afd379942ea887648a5de383571 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 7, Last appended: 1.7, Last appended by leader: 7, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "cc8526b37e1a4e17aa4785a4a8f15156" member_type: VOTER last_known_addr { host: "127.6.234.254" port: 39029 } } peers { permanent_uuid: "4312e033271940b593bea0332e185bff" member_type: VOTER last_known_addr { host: "127.6.234.253" port: 41509 } attrs { promote: false } } peers { permanent_uuid: "ea509afd379942ea887648a5de383571" member_type: NON_VOTER last_known_addr { host: "127.6.234.252" port: 39551 } attrs { promote: true } }
I20260620 11:42:53.429755 14005 sys_catalog.cc:455] T 00000000000000000000000000000000 P ea509afd379942ea887648a5de383571 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 committed_config { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "cc8526b37e1a4e17aa4785a4a8f15156" member_type: VOTER last_known_addr { host: "127.6.234.254" port: 39029 } } peers { permanent_uuid: "4312e033271940b593bea0332e185bff" member_type: VOTER last_known_addr { host: "127.6.234.253" port: 41509 } attrs { promote: false } } peers { permanent_uuid: "ea509afd379942ea887648a5de383571" member_type: NON_VOTER last_known_addr { host: "127.6.234.252" port: 39551 } attrs { promote: true } } }
I20260620 11:42:53.429819 14002 sys_catalog.cc:564] T 00000000000000000000000000000000 P ea509afd379942ea887648a5de383571 [sys.catalog]: configured and running, proceeding with master startup.
I20260620 11:42:53.429883 14005 sys_catalog.cc:458] T 00000000000000000000000000000000 P ea509afd379942ea887648a5de383571 [sys.catalog]: This master's current role is: LEARNER
I20260620 11:42:53.432065 14016 catalog_manager.cc:1259] Loaded cluster ID: 62ddcf685f6646508efb7ea8accaf0cf
I20260620 11:42:53.432129 14016 catalog_manager.cc:1552] T 00000000000000000000000000000000 P ea509afd379942ea887648a5de383571: loading cluster ID for follower catalog manager: success
I20260620 11:42:53.432848 14016 catalog_manager.cc:1574] T 00000000000000000000000000000000 P ea509afd379942ea887648a5de383571: acquiring CA information for follower catalog manager: success
I20260620 11:42:53.433115 14016 catalog_manager.cc:1602] T 00000000000000000000000000000000 P ea509afd379942ea887648a5de383571: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
W20260620 11:42:53.480196 13920 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 11:42:53.480429 13920 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 11:42:53.480469 13920 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 11:42:53.481631 13920 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 11:42:53.481694 13920 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 11:42:53.481761 13920 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.6.234.193
I20260620 11:42:53.483271 13920 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.6.234.212:38681
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task1JRg7N/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781955746769051-7083-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-task1JRg7N/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781955746769051-7083-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task1JRg7N/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781955746769051-7083-0/krb5kdc/oryx_127.6.234.193.keytab
--principal=oryx/127.6.234.193
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.6.234.193:38171
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task1JRg7N/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781955746769051-7083-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.6.234.193
--webserver_port=37865
--tserver_master_addrs=127.6.234.254:39029,127.6.234.253:41509,127.6.234.252:39551
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.13920
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.6.234.193
--log_dir=/tmp/dist-test-task1JRg7N/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781955746769051-7083-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 20 Jun 2026 10:32:37 UTC on ccycloud.quasar-bgyofv.root.comops.site
build id 4457173
I20260620 11:42:53.483512 13920 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 11:42:53.483747 13920 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 11:42:53.488569 14022 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 11:42:53.488579 14021 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 11:42:53.488691 14024 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 11:42:53.488910 13920 server_base.cc:1034] running on GCE node
I20260620 11:42:53.489105 13920 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 11:42:53.489356 13920 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 11:42:53.490520 13920 hybrid_clock.cc:648] HybridClock initialized: now 1781955773490505 us; error 30 us; skew 500 ppm
Jun 20 11:42:53 dist-test-slave-cjc6 krb5kdc[13130](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781955773, etypes {rep=17 tkt=17 ses=17}, oryx/127.6.234.193@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 11:42:53.493314 13920 init.cc:377] Logged in from keytab as oryx/127.6.234.193@KRBTEST.COM (short username oryx)
I20260620 11:42:53.493842 13920 webserver.cc:460] Webserver started at http://127.6.234.193:37865/ using document root <none> and password file <none>
I20260620 11:42:53.494091 13920 fs_manager.cc:359] Metadata directory not provided
I20260620 11:42:53.494158 13920 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 11:42:53.495936 13920 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.002s sys 0.000s
I20260620 11:42:53.496660 14031 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 11:42:53.496872 13920 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260620 11:42:53.496965 13920 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task1JRg7N/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781955746769051-7083-0/minicluster-data/ts-0/data,/tmp/dist-test-task1JRg7N/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781955746769051-7083-0/minicluster-data/ts-0/wal
uuid: "35c3aac7e3ce4bb3bb0294aa36fd45ab"
format_stamp: "Formatted at 2026-06-20 11:42:52 on dist-test-slave-cjc6"
server_key: "ad11f460cee7d38a2e66cf539338d192"
server_key_iv: "8ea088124828eb1f37680019cf3038bd"
server_key_version: "encryptionkey@0"
I20260620 11:42:53.497390 13920 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task1JRg7N/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781955746769051-7083-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-task1JRg7N/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781955746769051-7083-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-task1JRg7N/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781955746769051-7083-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 11:42:53.528348 13920 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 11:42:53.530936 13920 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 11:42:53.531054 13920 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 11:42:53.531332 13920 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 11:42:53.531670 13920 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 11:42:53.531731 13920 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 11:42:53.531774 13920 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 11:42:53.531800 13920 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 11:42:53.537549 13920 rpc_server.cc:307] RPC server started. Bound to: 127.6.234.193:38171
I20260620 11:42:53.537631 14144 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.6.234.193:38171 every 8 connection(s)
I20260620 11:42:53.538013 13920 server_base.cc:1166] Dumped server information to /tmp/dist-test-task1JRg7N/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781955746769051-7083-0/minicluster-data/ts-0/data/info.pb
I20260620 11:42:53.538906 7083 external_mini_cluster.cc:1388] Started /tmp/dist-test-task1JRg7N/build/release/bin/kudu as pid 13920
I20260620 11:42:53.539076 7083 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task1JRg7N/build/release/bin/kudu with pid 13371
I20260620 11:42:53.544670 7083 external_mini_cluster.cc:1326] Running /tmp/dist-test-task1JRg7N/build/release/bin/kudu
/tmp/dist-test-task1JRg7N/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task1JRg7N/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781955746769051-7083-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-task1JRg7N/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781955746769051-7083-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task1JRg7N/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781955746769051-7083-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-task1JRg7N/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781955746769051-7083-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.6.234.194:39263
--local_ip_for_outbound_sockets=127.6.234.194
--tserver_master_addrs=127.6.234.254:39029
--webserver_port=41999
--webserver_interface=127.6.234.194
--keytab_file=/tmp/dist-test-task1JRg7N/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781955746769051-7083-0/krb5kdc/oryx_127.6.234.194.keytab
--principal=oryx/127.6.234.194
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.6.234.212:38681
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_master_addrs=127.6.234.254:39029,127.6.234.253:41509
--tserver_master_addrs=127.6.234.254:39029,127.6.234.253:41509,127.6.234.252:39551 with env {KRB5CCNAME=/tmp/dist-test-task1JRg7N/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781955746769051-7083-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task1JRg7N/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781955746769051-7083-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task1JRg7N/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781955746769051-7083-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Jun 20 11:42:53 dist-test-slave-cjc6 krb5kdc[13130](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781955773, etypes {rep=17 tkt=17 ses=17}, oryx/127.6.234.193@KRBTEST.COM for oryx/127.6.234.253@KRBTEST.COM
Jun 20 11:42:53 dist-test-slave-cjc6 krb5kdc[13130](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781955773, etypes {rep=17 tkt=17 ses=17}, oryx/127.6.234.193@KRBTEST.COM for oryx/127.6.234.254@KRBTEST.COM
Jun 20 11:42:53 dist-test-slave-cjc6 krb5kdc[13130](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781955773, etypes {rep=17 tkt=17 ses=17}, oryx/127.6.234.193@KRBTEST.COM for oryx/127.6.234.252@KRBTEST.COM
I20260620 11:42:53.551573 13824 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 11:42:53.539585 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.6.234.193:35233 (local address 127.6.234.253:41509)
0620 11:42:53.539681 (+ 96us) server_negotiation.cc:206] Beginning negotiation
0620 11:42:53.539684 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0620 11:42:53.540734 (+ 1050us) server_negotiation.cc:407] Connection header received
0620 11:42:53.540785 (+ 51us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 11:42:53.540786 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 11:42:53.540799 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 11:42:53.540865 (+ 66us) server_negotiation.cc:226] Negotiated authn=SASL
0620 11:42:53.541367 (+ 502us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 11:42:53.542166 (+ 799us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 11:42:53.543728 (+ 1562us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 11:42:53.544054 (+ 326us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 11:42:53.544124 (+ 70us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 11:42:53.547526 (+ 3402us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 11:42:53.547527 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 11:42:53.547527 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 11:42:53.547535 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0620 11:42:53.549835 (+ 2300us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 11:42:53.550446 (+ 611us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 11:42:53.550448 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 11:42:53.550448 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 11:42:53.550512 (+ 64us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 11:42:53.550804 (+ 292us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 11:42:53.550805 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 11:42:53.550805 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 11:42:53.550966 (+ 161us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 11:42:53.551033 (+ 67us) server_negotiation.cc:1035] Waiting for connection context
0620 11:42:53.551386 (+ 353us) server_negotiation.cc:299] Negotiation successful
0620 11:42:53.551481 (+ 95us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":63}
I20260620 11:42:53.551888 14149 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 11:42:53.538812 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.234.254:39029 (local address 127.6.234.193:52845)
0620 11:42:53.538995 (+ 183us) negotiation.cc:107] Waiting for socket to connect
0620 11:42:53.539038 (+ 43us) client_negotiation.cc:174] Beginning negotiation
0620 11:42:53.540303 (+ 1265us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 11:42:53.540574 (+ 271us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 11:42:53.540580 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 11:42:53.540943 (+ 363us) client_negotiation.cc:189] Negotiated authn=SASL
0620 11:42:53.541259 (+ 316us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 11:42:53.541264 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 11:42:53.542305 (+ 1041us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 11:42:53.542306 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 11:42:53.543628 (+ 1322us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 11:42:53.543631 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 11:42:53.544102 (+ 471us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 11:42:53.544102 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 11:42:53.544238 (+ 136us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 11:42:53.544973 (+ 735us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 11:42:53.544978 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0620 11:42:53.547404 (+ 2426us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 11:42:53.549966 (+ 2562us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 11:42:53.549966 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 11:42:53.549967 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0620 11:42:53.550555 (+ 588us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 11:42:53.551007 (+ 452us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 11:42:53.551008 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 11:42:53.551008 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0620 11:42:53.551071 (+ 63us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 11:42:53.551367 (+ 296us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 11:42:53.551368 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 11:42:53.551402 (+ 34us) client_negotiation.cc:769] Sending connection context
0620 11:42:53.551476 (+ 74us) client_negotiation.cc:240] Negotiation successful
0620 11:42:53.551638 (+ 162us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":142,"mutex_wait_us":44,"thread_start_us":106,"threads_started":1}
I20260620 11:42:53.552208 13215 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 11:42:53.539124 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.6.234.193:52845 (local address 127.6.234.254:39029)
0620 11:42:53.539167 (+ 43us) server_negotiation.cc:206] Beginning negotiation
0620 11:42:53.539169 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 11:42:53.539178 (+ 9us) server_negotiation.cc:407] Connection header received
0620 11:42:53.540401 (+ 1223us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 11:42:53.540403 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 11:42:53.540415 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 11:42:53.540467 (+ 52us) server_negotiation.cc:226] Negotiated authn=SASL
0620 11:42:53.541367 (+ 900us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 11:42:53.542166 (+ 799us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 11:42:53.543740 (+ 1574us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 11:42:53.544001 (+ 261us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 11:42:53.544092 (+ 91us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 11:42:53.547511 (+ 3419us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 11:42:53.547513 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 11:42:53.547513 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 11:42:53.547520 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0620 11:42:53.549889 (+ 2369us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 11:42:53.550641 (+ 752us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 11:42:53.550642 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 11:42:53.550642 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 11:42:53.550700 (+ 58us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 11:42:53.551139 (+ 439us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 11:42:53.551140 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 11:42:53.551140 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 11:42:53.551277 (+ 137us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 11:42:53.551342 (+ 65us) server_negotiation.cc:1035] Waiting for connection context
0620 11:42:53.552026 (+ 684us) server_negotiation.cc:299] Negotiation successful
0620 11:42:53.552106 (+ 80us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":22}
I20260620 11:42:53.552603 14146 heartbeater.cc:347] Connected to a master server at 127.6.234.254:39029
I20260620 11:42:53.552714 14146 heartbeater.cc:464] Registering TS with master...
I20260620 11:42:53.552902 14150 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 11:42:53.539124 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.234.253:41509 (local address 127.6.234.193:35233)
0620 11:42:53.540643 (+ 1519us) negotiation.cc:107] Waiting for socket to connect
0620 11:42:53.540651 (+ 8us) client_negotiation.cc:174] Beginning negotiation
0620 11:42:53.540696 (+ 45us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 11:42:53.540929 (+ 233us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 11:42:53.540929 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 11:42:53.541088 (+ 159us) client_negotiation.cc:189] Negotiated authn=SASL
0620 11:42:53.541258 (+ 170us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 11:42:53.541264 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 11:42:53.542313 (+ 1049us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 11:42:53.542313 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 11:42:53.543628 (+ 1315us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 11:42:53.543631 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 11:42:53.544182 (+ 551us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 11:42:53.544184 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 11:42:53.544263 (+ 79us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 11:42:53.544942 (+ 679us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 11:42:53.544953 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0620 11:42:53.547429 (+ 2476us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 11:42:53.549937 (+ 2508us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 11:42:53.549944 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 11:42:53.549947 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0620 11:42:53.550329 (+ 382us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 11:42:53.550592 (+ 263us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 11:42:53.550593 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 11:42:53.550593 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0620 11:42:53.550738 (+ 145us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 11:42:53.551059 (+ 321us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 11:42:53.551066 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 11:42:53.551227 (+ 161us) client_negotiation.cc:769] Sending connection context
0620 11:42:53.551913 (+ 686us) client_negotiation.cc:240] Negotiation successful
0620 11:42:53.552730 (+ 817us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1496,"mutex_wait_us":72,"thread_start_us":62,"threads_started":1}
I20260620 11:42:53.553145 14151 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 11:42:53.538813 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.234.252:39551 (local address 127.6.234.193:45443)
0620 11:42:53.540492 (+ 1679us) negotiation.cc:107] Waiting for socket to connect
0620 11:42:53.540502 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0620 11:42:53.540556 (+ 54us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 11:42:53.544608 (+ 4052us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 11:42:53.544609 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 11:42:53.544822 (+ 213us) client_negotiation.cc:189] Negotiated authn=SASL
0620 11:42:53.544898 (+ 76us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 11:42:53.544900 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 11:42:53.546120 (+ 1220us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 11:42:53.546121 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 11:42:53.546996 (+ 875us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 11:42:53.547000 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 11:42:53.547520 (+ 520us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 11:42:53.547521 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 11:42:53.547594 (+ 73us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 11:42:53.548121 (+ 527us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 11:42:53.548126 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0620 11:42:53.549764 (+ 1638us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 11:42:53.551929 (+ 2165us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 11:42:53.551930 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 11:42:53.551931 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0620 11:42:53.552161 (+ 230us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 11:42:53.552410 (+ 249us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 11:42:53.552411 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 11:42:53.552411 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0620 11:42:53.552475 (+ 64us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 11:42:53.552773 (+ 298us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 11:42:53.552774 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 11:42:53.552813 (+ 39us) client_negotiation.cc:769] Sending connection context
0620 11:42:53.552858 (+ 45us) client_negotiation.cc:240] Negotiation successful
0620 11:42:53.552945 (+ 87us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1650,"mutex_wait_us":3,"thread_start_us":107,"threads_started":1}
I20260620 11:42:53.552934 14146 heartbeater.cc:511] Master 127.6.234.254:39029 requested a full tablet report, sending...
I20260620 11:42:53.553238 14152 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 11:42:53.539555 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.6.234.193:45443 (local address 127.6.234.252:39551)
0620 11:42:53.544472 (+ 4917us) server_negotiation.cc:206] Beginning negotiation
0620 11:42:53.544474 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 11:42:53.544482 (+ 8us) server_negotiation.cc:407] Connection header received
0620 11:42:53.544503 (+ 21us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 11:42:53.544503 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 11:42:53.544514 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 11:42:53.544591 (+ 77us) server_negotiation.cc:226] Negotiated authn=SASL
0620 11:42:53.544996 (+ 405us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 11:42:53.546014 (+ 1018us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 11:42:53.547098 (+ 1084us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 11:42:53.547428 (+ 330us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 11:42:53.547742 (+ 314us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 11:42:53.549866 (+ 2124us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 11:42:53.549868 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 11:42:53.549868 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 11:42:53.549875 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0620 11:42:53.551821 (+ 1946us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 11:42:53.552255 (+ 434us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 11:42:53.552256 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 11:42:53.552257 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 11:42:53.552319 (+ 62us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 11:42:53.552553 (+ 234us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 11:42:53.552554 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 11:42:53.552554 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 11:42:53.552690 (+ 136us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 11:42:53.553001 (+ 311us) server_negotiation.cc:1035] Waiting for connection context
0620 11:42:53.553046 (+ 45us) server_negotiation.cc:299] Negotiation successful
0620 11:42:53.553121 (+ 75us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":4880,"thread_start_us":60,"threads_started":1}
I20260620 11:42:53.553785 13175 ts_manager.cc:194] Re-registered known tserver with Master: 35c3aac7e3ce4bb3bb0294aa36fd45ab (127.6.234.193:38171)
I20260620 11:42:53.553862 14147 heartbeater.cc:347] Connected to a master server at 127.6.234.253:41509
I20260620 11:42:53.553910 14147 heartbeater.cc:464] Registering TS with master...
I20260620 11:42:53.553926 14145 heartbeater.cc:347] Connected to a master server at 127.6.234.252:39551
I20260620 11:42:53.553966 14145 heartbeater.cc:464] Registering TS with master...
I20260620 11:42:53.553978 14147 heartbeater.cc:511] Master 127.6.234.253:41509 requested a full tablet report, sending...
I20260620 11:42:53.554025 14145 heartbeater.cc:511] Master 127.6.234.252:39551 requested a full tablet report, sending...
I20260620 11:42:53.554255 13175 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.6.234.193@KRBTEST.COM'} at 127.6.234.193:52845
I20260620 11:42:53.554380 13966 ts_manager.cc:194] Registered new tserver with Master: 35c3aac7e3ce4bb3bb0294aa36fd45ab (127.6.234.193:38171)
I20260620 11:42:53.554423 13773 ts_manager.cc:194] Registered new tserver with Master: 35c3aac7e3ce4bb3bb0294aa36fd45ab (127.6.234.193:38171)
W20260620 11:42:53.627004 14153 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 11:42:53.627192 14153 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 11:42:53.627215 14153 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 11:42:53.628394 14153 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 11:42:53.628439 14153 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 11:42:53.628466 14153 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.6.234.194
I20260620 11:42:53.630230 14153 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.6.234.212:38681
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task1JRg7N/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781955746769051-7083-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-task1JRg7N/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781955746769051-7083-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task1JRg7N/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781955746769051-7083-0/krb5kdc/oryx_127.6.234.194.keytab
--principal=oryx/127.6.234.194
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.6.234.194:39263
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task1JRg7N/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781955746769051-7083-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.6.234.194
--webserver_port=41999
--tserver_master_addrs=127.6.234.254:39029,127.6.234.253:41509,127.6.234.252:39551
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.14153
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.6.234.194
--log_dir=/tmp/dist-test-task1JRg7N/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781955746769051-7083-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 20 Jun 2026 10:32:37 UTC on ccycloud.quasar-bgyofv.root.comops.site
build id 4457173
I20260620 11:42:53.630489 14153 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 11:42:53.630745 14153 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 11:42:53.635625 14159 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 11:42:53.635613 14158 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 11:42:53.635602 14161 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 11:42:53.636006 14153 server_base.cc:1034] running on GCE node
I20260620 11:42:53.636214 14153 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 11:42:53.636446 14153 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 11:42:53.637614 14153 hybrid_clock.cc:648] HybridClock initialized: now 1781955773637600 us; error 26 us; skew 500 ppm
Jun 20 11:42:53 dist-test-slave-cjc6 krb5kdc[13130](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781955773, etypes {rep=17 tkt=17 ses=17}, oryx/127.6.234.194@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 11:42:53.640069 14153 init.cc:377] Logged in from keytab as oryx/127.6.234.194@KRBTEST.COM (short username oryx)
I20260620 11:42:53.640584 14153 webserver.cc:460] Webserver started at http://127.6.234.194:41999/ using document root <none> and password file <none>
I20260620 11:42:53.640873 14153 fs_manager.cc:359] Metadata directory not provided
I20260620 11:42:53.640961 14153 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 11:42:53.642369 14153 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.000s sys 0.002s
I20260620 11:42:53.642958 14168 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 11:42:53.643143 14153 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.001s
I20260620 11:42:53.643237 14153 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task1JRg7N/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781955746769051-7083-0/minicluster-data/ts-1/data,/tmp/dist-test-task1JRg7N/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781955746769051-7083-0/minicluster-data/ts-1/wal
uuid: "553457094e274e55ace2e66d211e3171"
format_stamp: "Formatted at 2026-06-20 11:42:52 on dist-test-slave-cjc6"
server_key: "f8874bb3589fc5275bc848d8f9df445f"
server_key_iv: "75a5b4f95369ca6d8a94c9ea8b643d10"
server_key_version: "encryptionkey@0"
I20260620 11:42:53.643575 14153 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task1JRg7N/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781955746769051-7083-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-task1JRg7N/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781955746769051-7083-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-task1JRg7N/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781955746769051-7083-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 11:42:53.680998 14153 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 11:42:53.682870 14153 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 11:42:53.683040 14153 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 11:42:53.683282 14153 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 11:42:53.683596 14153 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 11:42:53.683651 14153 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 11:42:53.683694 14153 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 11:42:53.683722 14153 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 11:42:53.689610 14153 rpc_server.cc:307] RPC server started. Bound to: 127.6.234.194:39263
I20260620 11:42:53.689652 14281 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.6.234.194:39263 every 8 connection(s)
I20260620 11:42:53.690037 14153 server_base.cc:1166] Dumped server information to /tmp/dist-test-task1JRg7N/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781955746769051-7083-0/minicluster-data/ts-1/data/info.pb
I20260620 11:42:53.690495 7083 external_mini_cluster.cc:1388] Started /tmp/dist-test-task1JRg7N/build/release/bin/kudu as pid 14153
I20260620 11:42:53.690666 7083 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task1JRg7N/build/release/bin/kudu with pid 13505
I20260620 11:42:53.696466 7083 external_mini_cluster.cc:1326] Running /tmp/dist-test-task1JRg7N/build/release/bin/kudu
/tmp/dist-test-task1JRg7N/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task1JRg7N/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781955746769051-7083-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-task1JRg7N/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781955746769051-7083-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task1JRg7N/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781955746769051-7083-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-task1JRg7N/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781955746769051-7083-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.6.234.195:39097
--local_ip_for_outbound_sockets=127.6.234.195
--tserver_master_addrs=127.6.234.254:39029
--webserver_port=35099
--webserver_interface=127.6.234.195
--keytab_file=/tmp/dist-test-task1JRg7N/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781955746769051-7083-0/krb5kdc/oryx_127.6.234.195.keytab
--principal=oryx/127.6.234.195
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.6.234.212:38681
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_master_addrs=127.6.234.254:39029,127.6.234.253:41509
--tserver_master_addrs=127.6.234.254:39029,127.6.234.253:41509,127.6.234.252:39551 with env {KRB5CCNAME=/tmp/dist-test-task1JRg7N/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781955746769051-7083-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task1JRg7N/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781955746769051-7083-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task1JRg7N/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781955746769051-7083-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Jun 20 11:42:53 dist-test-slave-cjc6 krb5kdc[13130](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781955773, etypes {rep=17 tkt=17 ses=17}, oryx/127.6.234.194@KRBTEST.COM for oryx/127.6.234.254@KRBTEST.COM
Jun 20 11:42:53 dist-test-slave-cjc6 krb5kdc[13130](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781955773, etypes {rep=17 tkt=17 ses=17}, oryx/127.6.234.194@KRBTEST.COM for oryx/127.6.234.252@KRBTEST.COM
Jun 20 11:42:53 dist-test-slave-cjc6 krb5kdc[13130](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781955773, etypes {rep=17 tkt=17 ses=17}, oryx/127.6.234.194@KRBTEST.COM for oryx/127.6.234.253@KRBTEST.COM
I20260620 11:42:53.703790 13215 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 11:42:53.691774 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.6.234.194:53781 (local address 127.6.234.254:39029)
0620 11:42:53.692538 (+ 764us) server_negotiation.cc:206] Beginning negotiation
0620 11:42:53.692540 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 11:42:53.692549 (+ 9us) server_negotiation.cc:407] Connection header received
0620 11:42:53.692576 (+ 27us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 11:42:53.692578 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 11:42:53.692588 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 11:42:53.692638 (+ 50us) server_negotiation.cc:226] Negotiated authn=SASL
0620 11:42:53.693454 (+ 816us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 11:42:53.694092 (+ 638us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 11:42:53.695762 (+ 1670us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 11:42:53.696063 (+ 301us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 11:42:53.696107 (+ 44us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 11:42:53.700007 (+ 3900us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 11:42:53.700008 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 11:42:53.700009 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 11:42:53.700015 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0620 11:42:53.701873 (+ 1858us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 11:42:53.702488 (+ 615us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 11:42:53.702489 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 11:42:53.702490 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 11:42:53.702549 (+ 59us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 11:42:53.702860 (+ 311us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 11:42:53.702861 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 11:42:53.702861 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 11:42:53.703001 (+ 140us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 11:42:53.703035 (+ 34us) server_negotiation.cc:1035] Waiting for connection context
0620 11:42:53.703609 (+ 574us) server_negotiation.cc:299] Negotiation successful
0620 11:42:53.703691 (+ 82us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":731}
I20260620 11:42:53.703785 14152 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 11:42:53.691290 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.6.234.194:56129 (local address 127.6.234.252:39551)
0620 11:42:53.691335 (+ 45us) server_negotiation.cc:206] Beginning negotiation
0620 11:42:53.691337 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 11:42:53.692223 (+ 886us) server_negotiation.cc:407] Connection header received
0620 11:42:53.692345 (+ 122us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 11:42:53.692347 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 11:42:53.692364 (+ 17us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 11:42:53.692463 (+ 99us) server_negotiation.cc:226] Negotiated authn=SASL
0620 11:42:53.693465 (+ 1002us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 11:42:53.694337 (+ 872us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 11:42:53.695773 (+ 1436us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 11:42:53.696106 (+ 333us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 11:42:53.696148 (+ 42us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 11:42:53.699494 (+ 3346us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 11:42:53.699495 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 11:42:53.699496 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 11:42:53.699504 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0620 11:42:53.702339 (+ 2835us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 11:42:53.702807 (+ 468us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 11:42:53.702808 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 11:42:53.702808 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 11:42:53.702866 (+ 58us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 11:42:53.703093 (+ 227us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 11:42:53.703094 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 11:42:53.703094 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 11:42:53.703232 (+ 138us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 11:42:53.703303 (+ 71us) server_negotiation.cc:1035] Waiting for connection context
0620 11:42:53.703609 (+ 306us) server_negotiation.cc:299] Negotiation successful
0620 11:42:53.703691 (+ 82us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":21}
I20260620 11:42:53.703859 13824 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 11:42:53.692478 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.6.234.194:38157 (local address 127.6.234.253:41509)
0620 11:42:53.692522 (+ 44us) server_negotiation.cc:206] Beginning negotiation
0620 11:42:53.692524 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 11:42:53.694236 (+ 1712us) server_negotiation.cc:407] Connection header received
0620 11:42:53.694393 (+ 157us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 11:42:53.694395 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 11:42:53.694408 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 11:42:53.694437 (+ 29us) server_negotiation.cc:226] Negotiated authn=SASL
0620 11:42:53.694890 (+ 453us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 11:42:53.695637 (+ 747us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 11:42:53.697175 (+ 1538us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 11:42:53.697458 (+ 283us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 11:42:53.697502 (+ 44us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 11:42:53.699983 (+ 2481us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 11:42:53.699985 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 11:42:53.699986 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 11:42:53.699993 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0620 11:42:53.702073 (+ 2080us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 11:42:53.702512 (+ 439us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 11:42:53.702513 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 11:42:53.702513 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 11:42:53.702562 (+ 49us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 11:42:53.702860 (+ 298us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 11:42:53.702861 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 11:42:53.702861 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 11:42:53.703174 (+ 313us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 11:42:53.703247 (+ 73us) server_negotiation.cc:1035] Waiting for connection context
0620 11:42:53.703634 (+ 387us) server_negotiation.cc:299] Negotiation successful
0620 11:42:53.703722 (+ 88us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":19}
I20260620 11:42:53.704548 14286 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 11:42:53.690808 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.234.254:39029 (local address 127.6.234.194:53781)
0620 11:42:53.691344 (+ 536us) negotiation.cc:107] Waiting for socket to connect
0620 11:42:53.691405 (+ 61us) client_negotiation.cc:174] Beginning negotiation
0620 11:42:53.692107 (+ 702us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 11:42:53.692985 (+ 878us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 11:42:53.692986 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 11:42:53.693206 (+ 220us) client_negotiation.cc:189] Negotiated authn=SASL
0620 11:42:53.693343 (+ 137us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 11:42:53.693349 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 11:42:53.694220 (+ 871us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 11:42:53.694221 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 11:42:53.695643 (+ 1422us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 11:42:53.695645 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 11:42:53.696194 (+ 549us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 11:42:53.696195 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 11:42:53.696331 (+ 136us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 11:42:53.696975 (+ 644us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 11:42:53.696986 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0620 11:42:53.699916 (+ 2930us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 11:42:53.701977 (+ 2061us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 11:42:53.701983 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 11:42:53.701985 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0620 11:42:53.702397 (+ 412us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 11:42:53.702628 (+ 231us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 11:42:53.702629 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 11:42:53.702629 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0620 11:42:53.702781 (+ 152us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 11:42:53.703299 (+ 518us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 11:42:53.703299 (+ 0us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 11:42:53.703433 (+ 134us) client_negotiation.cc:769] Sending connection context
0620 11:42:53.704218 (+ 785us) client_negotiation.cc:240] Negotiation successful
0620 11:42:53.704296 (+ 78us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":498,"thread_start_us":77,"threads_started":1}
I20260620 11:42:53.704550 14288 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 11:42:53.691074 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.234.253:41509 (local address 127.6.234.194:38157)
0620 11:42:53.694172 (+ 3098us) negotiation.cc:107] Waiting for socket to connect
0620 11:42:53.694181 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0620 11:42:53.694263 (+ 82us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 11:42:53.694496 (+ 233us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 11:42:53.694497 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 11:42:53.694720 (+ 223us) client_negotiation.cc:189] Negotiated authn=SASL
0620 11:42:53.694795 (+ 75us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 11:42:53.694797 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 11:42:53.695777 (+ 980us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 11:42:53.695778 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 11:42:53.697076 (+ 1298us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 11:42:53.697078 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 11:42:53.697579 (+ 501us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 11:42:53.697581 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 11:42:53.697649 (+ 68us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 11:42:53.698170 (+ 521us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 11:42:53.698176 (+ 6us) client_negotiation.cc:656] calling sasl_client_start()
0620 11:42:53.699882 (+ 1706us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 11:42:53.702162 (+ 2280us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 11:42:53.702163 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 11:42:53.702164 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0620 11:42:53.702430 (+ 266us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 11:42:53.702632 (+ 202us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 11:42:53.702632 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 11:42:53.702632 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0620 11:42:53.702776 (+ 144us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 11:42:53.703259 (+ 483us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 11:42:53.703265 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 11:42:53.703440 (+ 175us) client_negotiation.cc:769] Sending connection context
0620 11:42:53.703992 (+ 552us) client_negotiation.cc:240] Negotiation successful
0620 11:42:53.704189 (+ 197us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":3080,"thread_start_us":47,"threads_started":1}
I20260620 11:42:53.704550 14287 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 11:42:53.690869 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.234.252:39551 (local address 127.6.234.194:56129)
0620 11:42:53.692153 (+ 1284us) negotiation.cc:107] Waiting for socket to connect
0620 11:42:53.692164 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0620 11:42:53.692251 (+ 87us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 11:42:53.692600 (+ 349us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 11:42:53.692604 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 11:42:53.693048 (+ 444us) client_negotiation.cc:189] Negotiated authn=SASL
0620 11:42:53.693342 (+ 294us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 11:42:53.693348 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 11:42:53.694473 (+ 1125us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 11:42:53.694475 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 11:42:53.695644 (+ 1169us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 11:42:53.695647 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 11:42:53.696225 (+ 578us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 11:42:53.696226 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 11:42:53.696333 (+ 107us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 11:42:53.697052 (+ 719us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 11:42:53.697057 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0620 11:42:53.699396 (+ 2339us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 11:42:53.702454 (+ 3058us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 11:42:53.702455 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 11:42:53.702456 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0620 11:42:53.702699 (+ 243us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 11:42:53.702947 (+ 248us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 11:42:53.702948 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 11:42:53.702948 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0620 11:42:53.703012 (+ 64us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 11:42:53.703329 (+ 317us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 11:42:53.703329 (+ 0us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 11:42:53.703433 (+ 104us) client_negotiation.cc:769] Sending connection context
0620 11:42:53.704143 (+ 710us) client_negotiation.cc:240] Negotiation successful
0620 11:42:53.704242 (+ 99us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1270,"thread_start_us":75,"threads_started":1}
I20260620 11:42:53.705003 14284 heartbeater.cc:347] Connected to a master server at 127.6.234.253:41509
I20260620 11:42:53.705080 14284 heartbeater.cc:464] Registering TS with master...
I20260620 11:42:53.705260 14282 heartbeater.cc:347] Connected to a master server at 127.6.234.252:39551
I20260620 11:42:53.705281 14284 heartbeater.cc:511] Master 127.6.234.253:41509 requested a full tablet report, sending...
I20260620 11:42:53.705307 14282 heartbeater.cc:464] Registering TS with master...
I20260620 11:42:53.705381 14282 heartbeater.cc:511] Master 127.6.234.252:39551 requested a full tablet report, sending...
I20260620 11:42:53.705408 14283 heartbeater.cc:347] Connected to a master server at 127.6.234.254:39029
I20260620 11:42:53.705442 14283 heartbeater.cc:464] Registering TS with master...
I20260620 11:42:53.705502 14283 heartbeater.cc:511] Master 127.6.234.254:39029 requested a full tablet report, sending...
I20260620 11:42:53.705813 13966 ts_manager.cc:194] Registered new tserver with Master: 553457094e274e55ace2e66d211e3171 (127.6.234.194:39263)
I20260620 11:42:53.705823 13175 ts_manager.cc:194] Re-registered known tserver with Master: 553457094e274e55ace2e66d211e3171 (127.6.234.194:39263)
I20260620 11:42:53.706037 13773 ts_manager.cc:194] Registered new tserver with Master: 553457094e274e55ace2e66d211e3171 (127.6.234.194:39263)
I20260620 11:42:53.706283 13175 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.6.234.194@KRBTEST.COM'} at 127.6.234.194:53781
W20260620 11:42:53.780870 14289 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 11:42:53.781080 14289 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 11:42:53.781104 14289 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 11:42:53.782356 14289 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 11:42:53.782399 14289 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 11:42:53.782428 14289 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.6.234.195
I20260620 11:42:53.783874 14289 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.6.234.212:38681
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task1JRg7N/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781955746769051-7083-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-task1JRg7N/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781955746769051-7083-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task1JRg7N/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781955746769051-7083-0/krb5kdc/oryx_127.6.234.195.keytab
--principal=oryx/127.6.234.195
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.6.234.195:39097
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task1JRg7N/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781955746769051-7083-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.6.234.195
--webserver_port=35099
--tserver_master_addrs=127.6.234.254:39029,127.6.234.253:41509,127.6.234.252:39551
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.14289
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.6.234.195
--log_dir=/tmp/dist-test-task1JRg7N/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781955746769051-7083-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 20 Jun 2026 10:32:37 UTC on ccycloud.quasar-bgyofv.root.comops.site
build id 4457173
I20260620 11:42:53.784116 14289 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 11:42:53.784358 14289 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 11:42:53.789423 14295 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 11:42:53.789430 14297 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 11:42:53.789464 14294 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 11:42:53.789579 14289 server_base.cc:1034] running on GCE node
I20260620 11:42:53.789820 14289 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 11:42:53.790077 14289 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 11:42:53.791221 14289 hybrid_clock.cc:648] HybridClock initialized: now 1781955773791211 us; error 26 us; skew 500 ppm
Jun 20 11:42:53 dist-test-slave-cjc6 krb5kdc[13130](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781955773, etypes {rep=17 tkt=17 ses=17}, oryx/127.6.234.195@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 11:42:53.793833 14289 init.cc:377] Logged in from keytab as oryx/127.6.234.195@KRBTEST.COM (short username oryx)
I20260620 11:42:53.794296 14289 webserver.cc:460] Webserver started at http://127.6.234.195:35099/ using document root <none> and password file <none>
I20260620 11:42:53.794523 14289 fs_manager.cc:359] Metadata directory not provided
I20260620 11:42:53.794576 14289 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 11:42:53.796006 14289 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.001s sys 0.000s
I20260620 11:42:53.796648 14304 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 11:42:53.796846 14289 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260620 11:42:53.796941 14289 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task1JRg7N/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781955746769051-7083-0/minicluster-data/ts-2/data,/tmp/dist-test-task1JRg7N/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781955746769051-7083-0/minicluster-data/ts-2/wal
uuid: "212956ba6c57456f9d631b576e69d6a2"
format_stamp: "Formatted at 2026-06-20 11:42:52 on dist-test-slave-cjc6"
server_key: "4b372029a0757ec2ccfe20e0cd19ba6c"
server_key_iv: "8f99d304325e013005332a7f4be9f355"
server_key_version: "encryptionkey@0"
I20260620 11:42:53.797269 14289 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task1JRg7N/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781955746769051-7083-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-task1JRg7N/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781955746769051-7083-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-task1JRg7N/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781955746769051-7083-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 11:42:53.825451 14289 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 11:42:53.827868 14289 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 11:42:53.828033 14289 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 11:42:53.828301 14289 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 11:42:53.828655 14289 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 11:42:53.828696 14289 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 11:42:53.828719 14289 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 11:42:53.828732 14289 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 11:42:53.834905 14289 rpc_server.cc:307] RPC server started. Bound to: 127.6.234.195:39097
I20260620 11:42:53.834949 14417 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.6.234.195:39097 every 8 connection(s)
I20260620 11:42:53.835282 14289 server_base.cc:1166] Dumped server information to /tmp/dist-test-task1JRg7N/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781955746769051-7083-0/minicluster-data/ts-2/data/info.pb
I20260620 11:42:53.841687 7083 external_mini_cluster.cc:1388] Started /tmp/dist-test-task1JRg7N/build/release/bin/kudu as pid 14289
Jun 20 11:42:53 dist-test-slave-cjc6 krb5kdc[13130](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781955773, etypes {rep=17 tkt=17 ses=17}, oryx/127.6.234.195@KRBTEST.COM for oryx/127.6.234.253@KRBTEST.COM
Jun 20 11:42:53 dist-test-slave-cjc6 krb5kdc[13130](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781955773, etypes {rep=17 tkt=17 ses=17}, oryx/127.6.234.195@KRBTEST.COM for oryx/127.6.234.252@KRBTEST.COM
Jun 20 11:42:53 dist-test-slave-cjc6 krb5kdc[13130](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781955773, etypes {rep=17 tkt=17 ses=17}, oryx/127.6.234.195@KRBTEST.COM for oryx/127.6.234.254@KRBTEST.COM
I20260620 11:42:53.846078 14422 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 11:42:53.835982 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.234.253:41509 (local address 127.6.234.195:60497)
0620 11:42:53.836165 (+ 183us) negotiation.cc:107] Waiting for socket to connect
0620 11:42:53.836207 (+ 42us) client_negotiation.cc:174] Beginning negotiation
0620 11:42:53.836827 (+ 620us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 11:42:53.837107 (+ 280us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 11:42:53.837111 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 11:42:53.837763 (+ 652us) client_negotiation.cc:189] Negotiated authn=SASL
0620 11:42:53.837937 (+ 174us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 11:42:53.837942 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 11:42:53.838695 (+ 753us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 11:42:53.838696 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 11:42:53.839505 (+ 809us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 11:42:53.839507 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 11:42:53.839920 (+ 413us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 11:42:53.839921 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 11:42:53.840056 (+ 135us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 11:42:53.840561 (+ 505us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 11:42:53.840588 (+ 27us) client_negotiation.cc:656] calling sasl_client_start()
0620 11:42:53.842296 (+ 1708us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 11:42:53.844342 (+ 2046us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 11:42:53.844349 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 11:42:53.844351 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0620 11:42:53.844632 (+ 281us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 11:42:53.844839 (+ 207us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 11:42:53.844840 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 11:42:53.844841 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0620 11:42:53.844994 (+ 153us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 11:42:53.845318 (+ 324us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 11:42:53.845325 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 11:42:53.845486 (+ 161us) client_negotiation.cc:769] Sending connection context
0620 11:42:53.845595 (+ 109us) client_negotiation.cc:240] Negotiation successful
0620 11:42:53.845785 (+ 190us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":141,"mutex_wait_us":51,"thread_start_us":113,"threads_started":1}
I20260620 11:42:53.846561 13824 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 11:42:53.836213 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.6.234.195:60497 (local address 127.6.234.253:41509)
0620 11:42:53.836281 (+ 68us) server_negotiation.cc:206] Beginning negotiation
0620 11:42:53.836283 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 11:42:53.836294 (+ 11us) server_negotiation.cc:407] Connection header received
0620 11:42:53.836936 (+ 642us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 11:42:53.836937 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 11:42:53.836947 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 11:42:53.837027 (+ 80us) server_negotiation.cc:226] Negotiated authn=SASL
0620 11:42:53.838033 (+ 1006us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 11:42:53.838583 (+ 550us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 11:42:53.839625 (+ 1042us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 11:42:53.839822 (+ 197us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 11:42:53.839915 (+ 93us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 11:42:53.842431 (+ 2516us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 11:42:53.842432 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 11:42:53.842433 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 11:42:53.842440 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0620 11:42:53.844267 (+ 1827us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 11:42:53.844696 (+ 429us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 11:42:53.844697 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 11:42:53.844698 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 11:42:53.844756 (+ 58us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 11:42:53.845078 (+ 322us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 11:42:53.845079 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 11:42:53.845079 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 11:42:53.845223 (+ 144us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 11:42:53.845288 (+ 65us) server_negotiation.cc:1035] Waiting for connection context
0620 11:42:53.846400 (+ 1112us) server_negotiation.cc:299] Negotiation successful
0620 11:42:53.846472 (+ 72us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":41}
I20260620 11:42:53.846982 14420 heartbeater.cc:347] Connected to a master server at 127.6.234.253:41509
I20260620 11:42:53.847091 14420 heartbeater.cc:464] Registering TS with master...
I20260620 11:42:53.847287 14420 heartbeater.cc:511] Master 127.6.234.253:41509 requested a full tablet report, sending...
I20260620 11:42:53.847301 13215 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 11:42:53.835974 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.6.234.195:36895 (local address 127.6.234.254:39029)
0620 11:42:53.836053 (+ 79us) server_negotiation.cc:206] Beginning negotiation
0620 11:42:53.836055 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 11:42:53.836533 (+ 478us) server_negotiation.cc:407] Connection header received
0620 11:42:53.837084 (+ 551us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 11:42:53.837085 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 11:42:53.837096 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 11:42:53.837144 (+ 48us) server_negotiation.cc:226] Negotiated authn=SASL
0620 11:42:53.838047 (+ 903us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 11:42:53.838815 (+ 768us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 11:42:53.839939 (+ 1124us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 11:42:53.840231 (+ 292us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 11:42:53.840271 (+ 40us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 11:42:53.843901 (+ 3630us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 11:42:53.843902 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 11:42:53.843902 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 11:42:53.843921 (+ 19us) server_negotiation.cc:922] calling sasl_server_start()
0620 11:42:53.845895 (+ 1974us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 11:42:53.846327 (+ 432us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 11:42:53.846328 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 11:42:53.846329 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 11:42:53.846384 (+ 55us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 11:42:53.846580 (+ 196us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 11:42:53.846581 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 11:42:53.846582 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 11:42:53.846714 (+ 132us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 11:42:53.846776 (+ 62us) server_negotiation.cc:1035] Waiting for connection context
0620 11:42:53.847152 (+ 376us) server_negotiation.cc:299] Negotiation successful
0620 11:42:53.847221 (+ 69us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":47}
I20260620 11:42:53.847400 14424 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 11:42:53.835981 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.234.254:39029 (local address 127.6.234.195:36895)
0620 11:42:53.836481 (+ 500us) negotiation.cc:107] Waiting for socket to connect
0620 11:42:53.836490 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0620 11:42:53.836828 (+ 338us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 11:42:53.837157 (+ 329us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 11:42:53.837158 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 11:42:53.837608 (+ 450us) client_negotiation.cc:189] Negotiated authn=SASL
0620 11:42:53.837937 (+ 329us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 11:42:53.837943 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 11:42:53.838907 (+ 964us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 11:42:53.838907 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 11:42:53.839838 (+ 931us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 11:42:53.839840 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 11:42:53.840349 (+ 509us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 11:42:53.840349 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 11:42:53.840434 (+ 85us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 11:42:53.841018 (+ 584us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 11:42:53.841023 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0620 11:42:53.843809 (+ 2786us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 11:42:53.845992 (+ 2183us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 11:42:53.845993 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 11:42:53.845993 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0620 11:42:53.846251 (+ 258us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 11:42:53.846453 (+ 202us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 11:42:53.846454 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 11:42:53.846454 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0620 11:42:53.846513 (+ 59us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 11:42:53.846805 (+ 292us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 11:42:53.846806 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 11:42:53.846848 (+ 42us) client_negotiation.cc:769] Sending connection context
0620 11:42:53.846926 (+ 78us) client_negotiation.cc:240] Negotiation successful
0620 11:42:53.847044 (+ 118us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":474,"mutex_wait_us":98,"thread_start_us":87,"threads_started":1}
I20260620 11:42:53.847766 13773 ts_manager.cc:194] Registered new tserver with Master: 212956ba6c57456f9d631b576e69d6a2 (127.6.234.195:39097)
I20260620 11:42:53.848203 14419 heartbeater.cc:347] Connected to a master server at 127.6.234.254:39029
I20260620 11:42:53.848251 14419 heartbeater.cc:464] Registering TS with master...
I20260620 11:42:53.848312 14419 heartbeater.cc:511] Master 127.6.234.254:39029 requested a full tablet report, sending...
I20260620 11:42:53.848439 14152 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 11:42:53.836426 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.6.234.195:54611 (local address 127.6.234.252:39551)
0620 11:42:53.836470 (+ 44us) server_negotiation.cc:206] Beginning negotiation
0620 11:42:53.836472 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 11:42:53.837077 (+ 605us) server_negotiation.cc:407] Connection header received
0620 11:42:53.837110 (+ 33us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 11:42:53.837111 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 11:42:53.837122 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 11:42:53.837174 (+ 52us) server_negotiation.cc:226] Negotiated authn=SASL
0620 11:42:53.838047 (+ 873us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 11:42:53.838807 (+ 760us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 11:42:53.839940 (+ 1133us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 11:42:53.840237 (+ 297us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 11:42:53.840274 (+ 37us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 11:42:53.843576 (+ 3302us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 11:42:53.843577 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 11:42:53.843578 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 11:42:53.843586 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0620 11:42:53.845903 (+ 2317us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 11:42:53.847221 (+ 1318us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 11:42:53.847222 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 11:42:53.847222 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 11:42:53.847283 (+ 61us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 11:42:53.847755 (+ 472us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 11:42:53.847756 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 11:42:53.847757 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 11:42:53.847895 (+ 138us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 11:42:53.847953 (+ 58us) server_negotiation.cc:1035] Waiting for connection context
0620 11:42:53.848258 (+ 305us) server_negotiation.cc:299] Negotiation successful
0620 11:42:53.848339 (+ 81us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":21}
I20260620 11:42:53.848541 13175 ts_manager.cc:194] Re-registered known tserver with Master: 212956ba6c57456f9d631b576e69d6a2 (127.6.234.195:39097)
I20260620 11:42:53.848999 13175 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.6.234.195@KRBTEST.COM'} at 127.6.234.195:36895
I20260620 11:42:53.851162 14423 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 11:42:53.836262 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.234.252:39551 (local address 127.6.234.195:54611)
0620 11:42:53.837004 (+ 742us) negotiation.cc:107] Waiting for socket to connect
0620 11:42:53.837013 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0620 11:42:53.837063 (+ 50us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 11:42:53.837223 (+ 160us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 11:42:53.837224 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 11:42:53.837631 (+ 407us) client_negotiation.cc:189] Negotiated authn=SASL
0620 11:42:53.837937 (+ 306us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 11:42:53.837943 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 11:42:53.838907 (+ 964us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 11:42:53.838907 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 11:42:53.839838 (+ 931us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 11:42:53.839840 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 11:42:53.840348 (+ 508us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 11:42:53.840349 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 11:42:53.840434 (+ 85us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 11:42:53.840927 (+ 493us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 11:42:53.840931 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0620 11:42:53.843486 (+ 2555us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 11:42:53.845989 (+ 2503us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 11:42:53.845989 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 11:42:53.845990 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0620 11:42:53.847142 (+ 1152us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 11:42:53.847616 (+ 474us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 11:42:53.847617 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 11:42:53.847617 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0620 11:42:53.847683 (+ 66us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 11:42:53.847955 (+ 272us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 11:42:53.847956 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 11:42:53.847995 (+ 39us) client_negotiation.cc:769] Sending connection context
0620 11:42:53.848060 (+ 65us) client_negotiation.cc:240] Negotiation successful
0620 11:42:53.848159 (+ 99us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":728,"mutex_wait_us":55,"thread_start_us":57,"threads_started":1}
I20260620 11:42:53.851622 14418 heartbeater.cc:347] Connected to a master server at 127.6.234.252:39551
I20260620 11:42:53.851682 14418 heartbeater.cc:464] Registering TS with master...
I20260620 11:42:53.851753 14418 heartbeater.cc:511] Master 127.6.234.252:39551 requested a full tablet report, sending...
I20260620 11:42:53.852051 13966 ts_manager.cc:194] Registered new tserver with Master: 212956ba6c57456f9d631b576e69d6a2 (127.6.234.195:39097)
I20260620 11:42:53.852808 14430 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 11:42:53.842946 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51634 (local address 127.6.234.254:39029)
0620 11:42:53.843157 (+ 211us) server_negotiation.cc:206] Beginning negotiation
0620 11:42:53.843159 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 11:42:53.843166 (+ 7us) server_negotiation.cc:407] Connection header received
0620 11:42:53.843186 (+ 20us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 11:42:53.843187 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 11:42:53.843200 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 11:42:53.844048 (+ 848us) server_negotiation.cc:226] Negotiated authn=SASL
0620 11:42:53.844085 (+ 37us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 11:42:53.844829 (+ 744us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 11:42:53.846018 (+ 1189us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 11:42:53.846381 (+ 363us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 11:42:53.849114 (+ 2733us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 11:42:53.849168 (+ 54us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 11:42:53.849169 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 11:42:53.849169 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 11:42:53.849174 (+ 5us) server_negotiation.cc:922] calling sasl_server_start()
0620 11:42:53.850750 (+ 1576us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 11:42:53.851164 (+ 414us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 11:42:53.851165 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 11:42:53.851166 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 11:42:53.851224 (+ 58us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 11:42:53.851871 (+ 647us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 11:42:53.851872 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 11:42:53.851872 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 11:42:53.852338 (+ 466us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 11:42:53.852483 (+ 145us) server_negotiation.cc:1035] Waiting for connection context
0620 11:42:53.852588 (+ 105us) server_negotiation.cc:299] Negotiation successful
0620 11:42:53.852667 (+ 79us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":182,"thread_start_us":76,"threads_started":1}
I20260620 11:42:53.854386 13824 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 11:42:53.843956 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38368 (local address 127.6.234.253:41509)
0620 11:42:53.846626 (+ 2670us) server_negotiation.cc:206] Beginning negotiation
0620 11:42:53.846628 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 11:42:53.846928 (+ 300us) server_negotiation.cc:407] Connection header received
0620 11:42:53.847031 (+ 103us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 11:42:53.847032 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 11:42:53.847044 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 11:42:53.847075 (+ 31us) server_negotiation.cc:226] Negotiated authn=SASL
0620 11:42:53.847877 (+ 802us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 11:42:53.848639 (+ 762us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 11:42:53.849634 (+ 995us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 11:42:53.849967 (+ 333us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 11:42:53.850004 (+ 37us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 11:42:53.851363 (+ 1359us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 11:42:53.851364 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 11:42:53.851364 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 11:42:53.851370 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0620 11:42:53.852948 (+ 1578us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 11:42:53.853479 (+ 531us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 11:42:53.853480 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 11:42:53.853480 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 11:42:53.853546 (+ 66us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 11:42:53.853802 (+ 256us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 11:42:53.853803 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 11:42:53.853803 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 11:42:53.853935 (+ 132us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 11:42:53.854031 (+ 96us) server_negotiation.cc:1035] Waiting for connection context
0620 11:42:53.854164 (+ 133us) server_negotiation.cc:299] Negotiation successful
0620 11:42:53.854245 (+ 81us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":2657,"thread_start_us":47,"threads_started":1}
I20260620 11:42:53.854488 13175 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:51634:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260620 11:42:53.856693 13175 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260620 11:42:53.858898 14433 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 11:42:53.844599 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43414 (local address 127.6.234.252:39551)
0620 11:42:53.846808 (+ 2209us) server_negotiation.cc:206] Beginning negotiation
0620 11:42:53.846811 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0620 11:42:53.851489 (+ 4678us) server_negotiation.cc:407] Connection header received
0620 11:42:53.851591 (+ 102us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 11:42:53.851592 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 11:42:53.851604 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 11:42:53.851639 (+ 35us) server_negotiation.cc:226] Negotiated authn=SASL
0620 11:42:53.852298 (+ 659us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 11:42:53.853066 (+ 768us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 11:42:53.854350 (+ 1284us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 11:42:53.854815 (+ 465us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 11:42:53.854894 (+ 79us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 11:42:53.855959 (+ 1065us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 11:42:53.855960 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 11:42:53.855960 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 11:42:53.855968 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0620 11:42:53.857561 (+ 1593us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 11:42:53.857944 (+ 383us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 11:42:53.857945 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 11:42:53.857946 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 11:42:53.858002 (+ 56us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 11:42:53.858217 (+ 215us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 11:42:53.858218 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 11:42:53.858218 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 11:42:53.858332 (+ 114us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 11:42:53.858442 (+ 110us) server_negotiation.cc:1035] Waiting for connection context
0620 11:42:53.858610 (+ 168us) server_negotiation.cc:299] Negotiation successful
0620 11:42:53.858714 (+ 104us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":2189,"thread_start_us":54,"threads_started":1}
I20260620 11:42:53.864847 14442 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 11:42:53.861089 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39722 (local address 127.6.234.195:39097)
0620 11:42:53.861617 (+ 528us) server_negotiation.cc:206] Beginning negotiation
0620 11:42:53.861621 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 11:42:53.861629 (+ 8us) server_negotiation.cc:407] Connection header received
0620 11:42:53.861657 (+ 28us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 11:42:53.861658 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 11:42:53.861670 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 11:42:53.861788 (+ 118us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 11:42:53.861914 (+ 126us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 11:42:53.862907 (+ 993us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 11:42:53.863959 (+ 1052us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 11:42:53.864447 (+ 488us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 11:42:53.864579 (+ 132us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 11:42:53.864664 (+ 85us) server_negotiation.cc:1035] Waiting for connection context
0620 11:42:53.864735 (+ 71us) server_negotiation.cc:299] Negotiation successful
0620 11:42:53.864774 (+ 39us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":487,"thread_start_us":75,"threads_started":1}
I20260620 11:42:53.864833 13733 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 11:42:53.860766 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.234.195:39097 (local address 127.0.0.1:39722)
0620 11:42:53.860933 (+ 167us) negotiation.cc:107] Waiting for socket to connect
0620 11:42:53.860944 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0620 11:42:53.860985 (+ 41us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 11:42:53.861786 (+ 801us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 11:42:53.861787 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 11:42:53.861788 (+ 1us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 11:42:53.861824 (+ 36us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 11:42:53.861825 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 11:42:53.863000 (+ 1175us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 11:42:53.863001 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 11:42:53.863860 (+ 859us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 11:42:53.863862 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 11:42:53.864592 (+ 730us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 11:42:53.864593 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 11:42:53.864653 (+ 60us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 11:42:53.864657 (+ 4us) client_negotiation.cc:769] Sending connection context
0620 11:42:53.864720 (+ 63us) client_negotiation.cc:240] Negotiation successful
0620 11:42:53.864760 (+ 40us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":149,"mutex_wait_us":1}
I20260620 11:42:53.865046 14439 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 11:42:53.860766 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.234.194:39263 (local address 127.0.0.1:49256)
0620 11:42:53.860997 (+ 231us) negotiation.cc:107] Waiting for socket to connect
0620 11:42:53.861008 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0620 11:42:53.861065 (+ 57us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 11:42:53.861285 (+ 220us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 11:42:53.861286 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 11:42:53.861288 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 11:42:53.861353 (+ 65us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 11:42:53.861354 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 11:42:53.862691 (+ 1337us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 11:42:53.862691 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 11:42:53.863881 (+ 1190us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 11:42:53.863883 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 11:42:53.864789 (+ 906us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 11:42:53.864790 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 11:42:53.864872 (+ 82us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 11:42:53.864875 (+ 3us) client_negotiation.cc:769] Sending connection context
0620 11:42:53.864906 (+ 31us) client_negotiation.cc:240] Negotiation successful
0620 11:42:53.864952 (+ 46us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":208,"mutex_wait_us":100,"thread_start_us":100,"threads_started":1}
I20260620 11:42:53.865355 14441 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 11:42:53.860940 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.234.193:38171 (local address 127.0.0.1:35936)
0620 11:42:53.861150 (+ 210us) negotiation.cc:107] Waiting for socket to connect
0620 11:42:53.861158 (+ 8us) client_negotiation.cc:174] Beginning negotiation
0620 11:42:53.861193 (+ 35us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 11:42:53.861770 (+ 577us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 11:42:53.861771 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 11:42:53.861774 (+ 3us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 11:42:53.861821 (+ 47us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 11:42:53.861824 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 11:42:53.862648 (+ 824us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 11:42:53.862649 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 11:42:53.863912 (+ 1263us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 11:42:53.863914 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 11:42:53.864875 (+ 961us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 11:42:53.864875 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 11:42:53.865201 (+ 326us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 11:42:53.865206 (+ 5us) client_negotiation.cc:769] Sending connection context
0620 11:42:53.865256 (+ 50us) client_negotiation.cc:240] Negotiation successful
0620 11:42:53.865291 (+ 35us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":195,"mutex_wait_us":28,"thread_start_us":88,"threads_started":1}
I20260620 11:42:53.865625 14443 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 11:42:53.861155 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35936 (local address 127.6.234.193:38171)
0620 11:42:53.861343 (+ 188us) server_negotiation.cc:206] Beginning negotiation
0620 11:42:53.861348 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 11:42:53.861358 (+ 10us) server_negotiation.cc:407] Connection header received
0620 11:42:53.861400 (+ 42us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 11:42:53.861403 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 11:42:53.861418 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 11:42:53.861498 (+ 80us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 11:42:53.861885 (+ 387us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 11:42:53.862557 (+ 672us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 11:42:53.863996 (+ 1439us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 11:42:53.864696 (+ 700us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 11:42:53.864757 (+ 61us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 11:42:53.864844 (+ 87us) server_negotiation.cc:1035] Waiting for connection context
0620 11:42:53.865496 (+ 652us) server_negotiation.cc:299] Negotiation successful
0620 11:42:53.865547 (+ 51us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":156,"thread_start_us":70,"threads_started":1}
I20260620 11:42:53.865631 14352 tablet_service.cc:1467] Processing CreateTablet for tablet bbbf5f0125f343c8a6166cde663ebee9 (DEFAULT_TABLE table=test-table [id=8e4bd3659a8c4596bc87119dd0a0feb9]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 11:42:53.866026 14352 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet bbbf5f0125f343c8a6166cde663ebee9. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 11:42:53.866356 14079 tablet_service.cc:1467] Processing CreateTablet for tablet bbbf5f0125f343c8a6166cde663ebee9 (DEFAULT_TABLE table=test-table [id=8e4bd3659a8c4596bc87119dd0a0feb9]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 11:42:53.866621 14079 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet bbbf5f0125f343c8a6166cde663ebee9. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 11:42:53.868803 14444 tablet_bootstrap.cc:492] T bbbf5f0125f343c8a6166cde663ebee9 P 212956ba6c57456f9d631b576e69d6a2: Bootstrap starting.
I20260620 11:42:53.868840 14440 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 11:42:53.860904 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49256 (local address 127.6.234.194:39263)
0620 11:42:53.861099 (+ 195us) server_negotiation.cc:206] Beginning negotiation
0620 11:42:53.861105 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0620 11:42:53.861115 (+ 10us) server_negotiation.cc:407] Connection header received
0620 11:42:53.861160 (+ 45us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 11:42:53.861163 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 11:42:53.861181 (+ 18us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 11:42:53.861428 (+ 247us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 11:42:53.861505 (+ 77us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 11:42:53.862584 (+ 1079us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 11:42:53.863983 (+ 1399us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 11:42:53.864682 (+ 699us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 11:42:53.865148 (+ 466us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 11:42:53.868700 (+ 3552us) server_negotiation.cc:1035] Waiting for connection context
0620 11:42:53.868728 (+ 28us) server_negotiation.cc:299] Negotiation successful
0620 11:42:53.868770 (+ 42us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":163,"thread_start_us":77,"threads_started":1}
I20260620 11:42:53.869483 14444 tablet_bootstrap.cc:654] T bbbf5f0125f343c8a6166cde663ebee9 P 212956ba6c57456f9d631b576e69d6a2: Neither blocks nor log segments found. Creating new log.
I20260620 11:42:53.869475 14445 tablet_bootstrap.cc:492] T bbbf5f0125f343c8a6166cde663ebee9 P 35c3aac7e3ce4bb3bb0294aa36fd45ab: Bootstrap starting.
I20260620 11:42:53.869503 14216 tablet_service.cc:1467] Processing CreateTablet for tablet bbbf5f0125f343c8a6166cde663ebee9 (DEFAULT_TABLE table=test-table [id=8e4bd3659a8c4596bc87119dd0a0feb9]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 11:42:53.869832 14444 log.cc:824] T bbbf5f0125f343c8a6166cde663ebee9 P 212956ba6c57456f9d631b576e69d6a2: Log is configured to *not* fsync() on all Append() calls
I20260620 11:42:53.869908 14216 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet bbbf5f0125f343c8a6166cde663ebee9. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 11:42:53.870107 14445 tablet_bootstrap.cc:654] T bbbf5f0125f343c8a6166cde663ebee9 P 35c3aac7e3ce4bb3bb0294aa36fd45ab: Neither blocks nor log segments found. Creating new log.
I20260620 11:42:53.870425 14445 log.cc:824] T bbbf5f0125f343c8a6166cde663ebee9 P 35c3aac7e3ce4bb3bb0294aa36fd45ab: Log is configured to *not* fsync() on all Append() calls
I20260620 11:42:53.870649 14444 tablet_bootstrap.cc:492] T bbbf5f0125f343c8a6166cde663ebee9 P 212956ba6c57456f9d631b576e69d6a2: No bootstrap required, opened a new log
I20260620 11:42:53.870725 14444 ts_tablet_manager.cc:1397] T bbbf5f0125f343c8a6166cde663ebee9 P 212956ba6c57456f9d631b576e69d6a2: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260620 11:42:53.871201 14445 tablet_bootstrap.cc:492] T bbbf5f0125f343c8a6166cde663ebee9 P 35c3aac7e3ce4bb3bb0294aa36fd45ab: No bootstrap required, opened a new log
I20260620 11:42:53.871279 14445 ts_tablet_manager.cc:1397] T bbbf5f0125f343c8a6166cde663ebee9 P 35c3aac7e3ce4bb3bb0294aa36fd45ab: Time spent bootstrapping tablet: real 0.002s user 0.000s sys 0.001s
I20260620 11:42:53.872789 14444 raft_consensus.cc:348] T bbbf5f0125f343c8a6166cde663ebee9 P 212956ba6c57456f9d631b576e69d6a2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "553457094e274e55ace2e66d211e3171" member_type: VOTER last_known_addr { host: "127.6.234.194" port: 39263 } } peers { permanent_uuid: "212956ba6c57456f9d631b576e69d6a2" member_type: VOTER last_known_addr { host: "127.6.234.195" port: 39097 } } peers { permanent_uuid: "35c3aac7e3ce4bb3bb0294aa36fd45ab" member_type: VOTER last_known_addr { host: "127.6.234.193" port: 38171 } }
I20260620 11:42:53.872938 14444 raft_consensus.cc:374] T bbbf5f0125f343c8a6166cde663ebee9 P 212956ba6c57456f9d631b576e69d6a2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 11:42:53.872983 14444 raft_consensus.cc:729] T bbbf5f0125f343c8a6166cde663ebee9 P 212956ba6c57456f9d631b576e69d6a2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 212956ba6c57456f9d631b576e69d6a2, State: Initialized, Role: FOLLOWER
I20260620 11:42:53.873116 14444 consensus_queue.cc:260] T bbbf5f0125f343c8a6166cde663ebee9 P 212956ba6c57456f9d631b576e69d6a2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "553457094e274e55ace2e66d211e3171" member_type: VOTER last_known_addr { host: "127.6.234.194" port: 39263 } } peers { permanent_uuid: "212956ba6c57456f9d631b576e69d6a2" member_type: VOTER last_known_addr { host: "127.6.234.195" port: 39097 } } peers { permanent_uuid: "35c3aac7e3ce4bb3bb0294aa36fd45ab" member_type: VOTER last_known_addr { host: "127.6.234.193" port: 38171 } }
I20260620 11:42:53.873373 14444 ts_tablet_manager.cc:1428] T bbbf5f0125f343c8a6166cde663ebee9 P 212956ba6c57456f9d631b576e69d6a2: Time spent starting tablet: real 0.003s user 0.003s sys 0.000s
I20260620 11:42:53.873473 14449 tablet_bootstrap.cc:492] T bbbf5f0125f343c8a6166cde663ebee9 P 553457094e274e55ace2e66d211e3171: Bootstrap starting.
I20260620 11:42:53.873966 14445 raft_consensus.cc:348] T bbbf5f0125f343c8a6166cde663ebee9 P 35c3aac7e3ce4bb3bb0294aa36fd45ab [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "553457094e274e55ace2e66d211e3171" member_type: VOTER last_known_addr { host: "127.6.234.194" port: 39263 } } peers { permanent_uuid: "212956ba6c57456f9d631b576e69d6a2" member_type: VOTER last_known_addr { host: "127.6.234.195" port: 39097 } } peers { permanent_uuid: "35c3aac7e3ce4bb3bb0294aa36fd45ab" member_type: VOTER last_known_addr { host: "127.6.234.193" port: 38171 } }
I20260620 11:42:53.874114 14445 raft_consensus.cc:374] T bbbf5f0125f343c8a6166cde663ebee9 P 35c3aac7e3ce4bb3bb0294aa36fd45ab [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 11:42:53.874135 14449 tablet_bootstrap.cc:654] T bbbf5f0125f343c8a6166cde663ebee9 P 553457094e274e55ace2e66d211e3171: Neither blocks nor log segments found. Creating new log.
I20260620 11:42:53.874158 14445 raft_consensus.cc:729] T bbbf5f0125f343c8a6166cde663ebee9 P 35c3aac7e3ce4bb3bb0294aa36fd45ab [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 35c3aac7e3ce4bb3bb0294aa36fd45ab, State: Initialized, Role: FOLLOWER
I20260620 11:42:53.874259 14419 heartbeater.cc:503] Master 127.6.234.254:39029 was elected leader, sending a full tablet report...
I20260620 11:42:53.874291 14445 consensus_queue.cc:260] T bbbf5f0125f343c8a6166cde663ebee9 P 35c3aac7e3ce4bb3bb0294aa36fd45ab [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "553457094e274e55ace2e66d211e3171" member_type: VOTER last_known_addr { host: "127.6.234.194" port: 39263 } } peers { permanent_uuid: "212956ba6c57456f9d631b576e69d6a2" member_type: VOTER last_known_addr { host: "127.6.234.195" port: 39097 } } peers { permanent_uuid: "35c3aac7e3ce4bb3bb0294aa36fd45ab" member_type: VOTER last_known_addr { host: "127.6.234.193" port: 38171 } }
I20260620 11:42:53.874460 14449 log.cc:824] T bbbf5f0125f343c8a6166cde663ebee9 P 553457094e274e55ace2e66d211e3171: Log is configured to *not* fsync() on all Append() calls
I20260620 11:42:53.874528 14445 ts_tablet_manager.cc:1428] T bbbf5f0125f343c8a6166cde663ebee9 P 35c3aac7e3ce4bb3bb0294aa36fd45ab: Time spent starting tablet: real 0.003s user 0.000s sys 0.003s
I20260620 11:42:53.874823 14146 heartbeater.cc:503] Master 127.6.234.254:39029 was elected leader, sending a full tablet report...
I20260620 11:42:53.876355 14449 tablet_bootstrap.cc:492] T bbbf5f0125f343c8a6166cde663ebee9 P 553457094e274e55ace2e66d211e3171: No bootstrap required, opened a new log
I20260620 11:42:53.876426 14449 ts_tablet_manager.cc:1397] T bbbf5f0125f343c8a6166cde663ebee9 P 553457094e274e55ace2e66d211e3171: Time spent bootstrapping tablet: real 0.003s user 0.001s sys 0.000s
I20260620 11:42:53.878252 14449 raft_consensus.cc:348] T bbbf5f0125f343c8a6166cde663ebee9 P 553457094e274e55ace2e66d211e3171 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "553457094e274e55ace2e66d211e3171" member_type: VOTER last_known_addr { host: "127.6.234.194" port: 39263 } } peers { permanent_uuid: "212956ba6c57456f9d631b576e69d6a2" member_type: VOTER last_known_addr { host: "127.6.234.195" port: 39097 } } peers { permanent_uuid: "35c3aac7e3ce4bb3bb0294aa36fd45ab" member_type: VOTER last_known_addr { host: "127.6.234.193" port: 38171 } }
I20260620 11:42:53.878368 14449 raft_consensus.cc:374] T bbbf5f0125f343c8a6166cde663ebee9 P 553457094e274e55ace2e66d211e3171 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 11:42:53.878389 14449 raft_consensus.cc:729] T bbbf5f0125f343c8a6166cde663ebee9 P 553457094e274e55ace2e66d211e3171 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 553457094e274e55ace2e66d211e3171, State: Initialized, Role: FOLLOWER
I20260620 11:42:53.878463 14449 consensus_queue.cc:260] T bbbf5f0125f343c8a6166cde663ebee9 P 553457094e274e55ace2e66d211e3171 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "553457094e274e55ace2e66d211e3171" member_type: VOTER last_known_addr { host: "127.6.234.194" port: 39263 } } peers { permanent_uuid: "212956ba6c57456f9d631b576e69d6a2" member_type: VOTER last_known_addr { host: "127.6.234.195" port: 39097 } } peers { permanent_uuid: "35c3aac7e3ce4bb3bb0294aa36fd45ab" member_type: VOTER last_known_addr { host: "127.6.234.193" port: 38171 } }
I20260620 11:42:53.878717 14449 ts_tablet_manager.cc:1428] T bbbf5f0125f343c8a6166cde663ebee9 P 553457094e274e55ace2e66d211e3171: Time spent starting tablet: real 0.002s user 0.001s sys 0.002s
I20260620 11:42:53.878754 14283 heartbeater.cc:503] Master 127.6.234.254:39029 was elected leader, sending a full tablet report...
I20260620 11:42:53.907581 14433 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 11:42:53.905086 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43426 (local address 127.6.234.252:39551)
0620 11:42:53.905148 (+ 62us) server_negotiation.cc:206] Beginning negotiation
0620 11:42:53.905150 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 11:42:53.905161 (+ 11us) server_negotiation.cc:407] Connection header received
0620 11:42:53.905188 (+ 27us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 11:42:53.905188 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 11:42:53.905209 (+ 21us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 11:42:53.905271 (+ 62us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 11:42:53.905412 (+ 141us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 11:42:53.906082 (+ 670us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 11:42:53.906925 (+ 843us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 11:42:53.907330 (+ 405us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 11:42:53.907387 (+ 57us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 11:42:53.907442 (+ 55us) server_negotiation.cc:1035] Waiting for connection context
0620 11:42:53.907505 (+ 63us) server_negotiation.cc:299] Negotiation successful
0620 11:42:53.907528 (+ 23us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":39}
I20260620 11:42:53.907589 14441 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 11:42:53.905004 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.234.252:39551 (local address 127.0.0.1:43426)
0620 11:42:53.905071 (+ 67us) negotiation.cc:107] Waiting for socket to connect
0620 11:42:53.905084 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0620 11:42:53.905143 (+ 59us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 11:42:53.905282 (+ 139us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 11:42:53.905283 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 11:42:53.905284 (+ 1us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 11:42:53.905336 (+ 52us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 11:42:53.905337 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 11:42:53.906154 (+ 817us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 11:42:53.906155 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 11:42:53.906852 (+ 697us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 11:42:53.906854 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 11:42:53.907404 (+ 550us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 11:42:53.907405 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 11:42:53.907450 (+ 45us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 11:42:53.907454 (+ 4us) client_negotiation.cc:769] Sending connection context
0620 11:42:53.907499 (+ 45us) client_negotiation.cc:240] Negotiation successful
0620 11:42:53.907534 (+ 35us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":44}
I20260620 11:42:53.907996 13976 raft_consensus.cc:1203] T 00000000000000000000000000000000 P ea509afd379942ea887648a5de383571 [term 1 LEARNER]: Deduplicated request from leader. Original: 1.6->[1.7-1.9] Dedup: 1.7->[1.8-1.9]
I20260620 11:42:53.908406 14005 sys_catalog.cc:455] T 00000000000000000000000000000000 P ea509afd379942ea887648a5de383571 [sys.catalog]: SysCatalogTable state changed. Reason: New leader cc8526b37e1a4e17aa4785a4a8f15156. Latest consensus state: current_term: 1 leader_uuid: "cc8526b37e1a4e17aa4785a4a8f15156" committed_config { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "cc8526b37e1a4e17aa4785a4a8f15156" member_type: VOTER last_known_addr { host: "127.6.234.254" port: 39029 } } peers { permanent_uuid: "4312e033271940b593bea0332e185bff" member_type: VOTER last_known_addr { host: "127.6.234.253" port: 41509 } attrs { promote: false } } peers { permanent_uuid: "ea509afd379942ea887648a5de383571" member_type: NON_VOTER last_known_addr { host: "127.6.234.252" port: 39551 } attrs { promote: true } } }
I20260620 11:42:53.908516 14005 sys_catalog.cc:458] T 00000000000000000000000000000000 P ea509afd379942ea887648a5de383571 [sys.catalog]: This master's current role is: LEARNER
W20260620 11:42:53.940876 14285 tablet.cc:2369] T bbbf5f0125f343c8a6166cde663ebee9 P 553457094e274e55ace2e66d211e3171: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260620 11:42:53.977974 14450 raft_consensus.cc:482] T bbbf5f0125f343c8a6166cde663ebee9 P 35c3aac7e3ce4bb3bb0294aa36fd45ab [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260620 11:42:53.978114 14450 raft_consensus.cc:504] T bbbf5f0125f343c8a6166cde663ebee9 P 35c3aac7e3ce4bb3bb0294aa36fd45ab [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "553457094e274e55ace2e66d211e3171" member_type: VOTER last_known_addr { host: "127.6.234.194" port: 39263 } } peers { permanent_uuid: "212956ba6c57456f9d631b576e69d6a2" member_type: VOTER last_known_addr { host: "127.6.234.195" port: 39097 } } peers { permanent_uuid: "35c3aac7e3ce4bb3bb0294aa36fd45ab" member_type: VOTER last_known_addr { host: "127.6.234.193" port: 38171 } }
I20260620 11:42:53.978531 14450 leader_election.cc:288] T bbbf5f0125f343c8a6166cde663ebee9 P 35c3aac7e3ce4bb3bb0294aa36fd45ab [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 553457094e274e55ace2e66d211e3171 (127.6.234.194:39263), 212956ba6c57456f9d631b576e69d6a2 (127.6.234.195:39097)
I20260620 11:42:53.982471 14442 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 11:42:53.978833 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.6.234.193:39283 (local address 127.6.234.195:39097)
0620 11:42:53.978880 (+ 47us) server_negotiation.cc:206] Beginning negotiation
0620 11:42:53.978882 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 11:42:53.978891 (+ 9us) server_negotiation.cc:407] Connection header received
0620 11:42:53.978924 (+ 33us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 11:42:53.978925 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 11:42:53.978935 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 11:42:53.979000 (+ 65us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 11:42:53.979226 (+ 226us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 11:42:53.979976 (+ 750us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 11:42:53.981515 (+ 1539us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 11:42:53.982105 (+ 590us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 11:42:53.982184 (+ 79us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 11:42:53.982191 (+ 7us) server_negotiation.cc:1035] Waiting for connection context
0620 11:42:53.982360 (+ 169us) server_negotiation.cc:299] Negotiation successful
0620 11:42:53.982399 (+ 39us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":24}
I20260620 11:42:53.982471 14440 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 11:42:53.978863 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.6.234.193:47927 (local address 127.6.234.194:39263)
0620 11:42:53.978909 (+ 46us) server_negotiation.cc:206] Beginning negotiation
0620 11:42:53.978911 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 11:42:53.978918 (+ 7us) server_negotiation.cc:407] Connection header received
0620 11:42:53.978944 (+ 26us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 11:42:53.978944 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 11:42:53.978953 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 11:42:53.979011 (+ 58us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 11:42:53.979226 (+ 215us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 11:42:53.979969 (+ 743us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 11:42:53.981516 (+ 1547us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 11:42:53.982089 (+ 573us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 11:42:53.982180 (+ 91us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 11:42:53.982191 (+ 11us) server_negotiation.cc:1035] Waiting for connection context
0620 11:42:53.982360 (+ 169us) server_negotiation.cc:299] Negotiation successful
0620 11:42:53.982398 (+ 38us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":24}
I20260620 11:42:53.982484 14151 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 11:42:53.978777 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.234.195:39097 (local address 127.6.234.193:39283)
0620 11:42:53.978827 (+ 50us) negotiation.cc:107] Waiting for socket to connect
0620 11:42:53.978835 (+ 8us) client_negotiation.cc:174] Beginning negotiation
0620 11:42:53.978865 (+ 30us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 11:42:53.979029 (+ 164us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 11:42:53.979030 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 11:42:53.979032 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 11:42:53.979143 (+ 111us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 11:42:53.979145 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 11:42:53.980059 (+ 914us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 11:42:53.980060 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 11:42:53.981404 (+ 1344us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 11:42:53.981415 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 11:42:53.982192 (+ 777us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 11:42:53.982193 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 11:42:53.982273 (+ 80us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 11:42:53.982279 (+ 6us) client_negotiation.cc:769] Sending connection context
0620 11:42:53.982341 (+ 62us) client_negotiation.cc:240] Negotiation successful
0620 11:42:53.982401 (+ 60us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":38,"mutex_wait_us":66}
I20260620 11:42:53.982496 14150 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 11:42:53.978729 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.234.194:39263 (local address 127.6.234.193:47927)
0620 11:42:53.978796 (+ 67us) negotiation.cc:107] Waiting for socket to connect
0620 11:42:53.978809 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0620 11:42:53.978860 (+ 51us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 11:42:53.979029 (+ 169us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 11:42:53.979030 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 11:42:53.979032 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 11:42:53.979143 (+ 111us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 11:42:53.979145 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 11:42:53.980060 (+ 915us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 11:42:53.980060 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 11:42:53.981404 (+ 1344us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 11:42:53.981408 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 11:42:53.982192 (+ 784us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 11:42:53.982193 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 11:42:53.982273 (+ 80us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 11:42:53.982279 (+ 6us) client_negotiation.cc:769] Sending connection context
0620 11:42:53.982341 (+ 62us) client_negotiation.cc:240] Negotiation successful
0620 11:42:53.982401 (+ 60us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":45,"mutex_wait_us":127}
I20260620 11:42:53.982950 14372 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bbbf5f0125f343c8a6166cde663ebee9" candidate_uuid: "35c3aac7e3ce4bb3bb0294aa36fd45ab" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "212956ba6c57456f9d631b576e69d6a2" is_pre_election: true
I20260620 11:42:53.983152 14372 raft_consensus.cc:2454] T bbbf5f0125f343c8a6166cde663ebee9 P 212956ba6c57456f9d631b576e69d6a2 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 35c3aac7e3ce4bb3bb0294aa36fd45ab in term 0.
I20260620 11:42:53.983126 14236 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bbbf5f0125f343c8a6166cde663ebee9" candidate_uuid: "35c3aac7e3ce4bb3bb0294aa36fd45ab" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "553457094e274e55ace2e66d211e3171" is_pre_election: true
I20260620 11:42:53.983251 14236 raft_consensus.cc:2454] T bbbf5f0125f343c8a6166cde663ebee9 P 553457094e274e55ace2e66d211e3171 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 35c3aac7e3ce4bb3bb0294aa36fd45ab in term 0.
I20260620 11:42:53.983422 14032 leader_election.cc:302] T bbbf5f0125f343c8a6166cde663ebee9 P 35c3aac7e3ce4bb3bb0294aa36fd45ab [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 212956ba6c57456f9d631b576e69d6a2, 35c3aac7e3ce4bb3bb0294aa36fd45ab; no voters:
I20260620 11:42:53.983574 14450 raft_consensus.cc:2781] T bbbf5f0125f343c8a6166cde663ebee9 P 35c3aac7e3ce4bb3bb0294aa36fd45ab [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260620 11:42:53.983661 14450 raft_consensus.cc:482] T bbbf5f0125f343c8a6166cde663ebee9 P 35c3aac7e3ce4bb3bb0294aa36fd45ab [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260620 11:42:53.983700 14450 raft_consensus.cc:3037] T bbbf5f0125f343c8a6166cde663ebee9 P 35c3aac7e3ce4bb3bb0294aa36fd45ab [term 0 FOLLOWER]: Advancing to term 1
I20260620 11:42:53.984459 14450 raft_consensus.cc:504] T bbbf5f0125f343c8a6166cde663ebee9 P 35c3aac7e3ce4bb3bb0294aa36fd45ab [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "553457094e274e55ace2e66d211e3171" member_type: VOTER last_known_addr { host: "127.6.234.194" port: 39263 } } peers { permanent_uuid: "212956ba6c57456f9d631b576e69d6a2" member_type: VOTER last_known_addr { host: "127.6.234.195" port: 39097 } } peers { permanent_uuid: "35c3aac7e3ce4bb3bb0294aa36fd45ab" member_type: VOTER last_known_addr { host: "127.6.234.193" port: 38171 } }
I20260620 11:42:53.984622 14450 leader_election.cc:288] T bbbf5f0125f343c8a6166cde663ebee9 P 35c3aac7e3ce4bb3bb0294aa36fd45ab [CANDIDATE]: Term 1 election: Requested vote from peers 553457094e274e55ace2e66d211e3171 (127.6.234.194:39263), 212956ba6c57456f9d631b576e69d6a2 (127.6.234.195:39097)
I20260620 11:42:53.984814 14236 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bbbf5f0125f343c8a6166cde663ebee9" candidate_uuid: "35c3aac7e3ce4bb3bb0294aa36fd45ab" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "553457094e274e55ace2e66d211e3171"
I20260620 11:42:53.984805 14372 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bbbf5f0125f343c8a6166cde663ebee9" candidate_uuid: "35c3aac7e3ce4bb3bb0294aa36fd45ab" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "212956ba6c57456f9d631b576e69d6a2"
I20260620 11:42:53.984894 14236 raft_consensus.cc:3037] T bbbf5f0125f343c8a6166cde663ebee9 P 553457094e274e55ace2e66d211e3171 [term 0 FOLLOWER]: Advancing to term 1
I20260620 11:42:53.984913 14372 raft_consensus.cc:3037] T bbbf5f0125f343c8a6166cde663ebee9 P 212956ba6c57456f9d631b576e69d6a2 [term 0 FOLLOWER]: Advancing to term 1
I20260620 11:42:53.985561 14236 raft_consensus.cc:2454] T bbbf5f0125f343c8a6166cde663ebee9 P 553457094e274e55ace2e66d211e3171 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 35c3aac7e3ce4bb3bb0294aa36fd45ab in term 1.
I20260620 11:42:53.985678 14372 raft_consensus.cc:2454] T bbbf5f0125f343c8a6166cde663ebee9 P 212956ba6c57456f9d631b576e69d6a2 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 35c3aac7e3ce4bb3bb0294aa36fd45ab in term 1.
I20260620 11:42:53.985790 14035 leader_election.cc:302] T bbbf5f0125f343c8a6166cde663ebee9 P 35c3aac7e3ce4bb3bb0294aa36fd45ab [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 35c3aac7e3ce4bb3bb0294aa36fd45ab, 553457094e274e55ace2e66d211e3171; no voters:
I20260620 11:42:53.985877 14450 raft_consensus.cc:2781] T bbbf5f0125f343c8a6166cde663ebee9 P 35c3aac7e3ce4bb3bb0294aa36fd45ab [term 1 FOLLOWER]: Leader election won for term 1
I20260620 11:42:53.986053 14450 raft_consensus.cc:686] T bbbf5f0125f343c8a6166cde663ebee9 P 35c3aac7e3ce4bb3bb0294aa36fd45ab [term 1 LEADER]: Becoming Leader. State: Replica: 35c3aac7e3ce4bb3bb0294aa36fd45ab, State: Running, Role: LEADER
I20260620 11:42:53.986168 14450 consensus_queue.cc:237] T bbbf5f0125f343c8a6166cde663ebee9 P 35c3aac7e3ce4bb3bb0294aa36fd45ab [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "553457094e274e55ace2e66d211e3171" member_type: VOTER last_known_addr { host: "127.6.234.194" port: 39263 } } peers { permanent_uuid: "212956ba6c57456f9d631b576e69d6a2" member_type: VOTER last_known_addr { host: "127.6.234.195" port: 39097 } } peers { permanent_uuid: "35c3aac7e3ce4bb3bb0294aa36fd45ab" member_type: VOTER last_known_addr { host: "127.6.234.193" port: 38171 } }
I20260620 11:42:53.987097 13174 catalog_manager.cc:5495] T bbbf5f0125f343c8a6166cde663ebee9 P 35c3aac7e3ce4bb3bb0294aa36fd45ab reported cstate change: term changed from 0 to 1, leader changed from <none> to 35c3aac7e3ce4bb3bb0294aa36fd45ab (127.6.234.193). New cstate: current_term: 1 leader_uuid: "35c3aac7e3ce4bb3bb0294aa36fd45ab" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "553457094e274e55ace2e66d211e3171" member_type: VOTER last_known_addr { host: "127.6.234.194" port: 39263 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "212956ba6c57456f9d631b576e69d6a2" member_type: VOTER last_known_addr { host: "127.6.234.195" port: 39097 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "35c3aac7e3ce4bb3bb0294aa36fd45ab" member_type: VOTER last_known_addr { host: "127.6.234.193" port: 38171 } health_report { overall_health: HEALTHY } } }
I20260620 11:42:53.987703 13734 sys_catalog.cc:455] T 00000000000000000000000000000000 P cc8526b37e1a4e17aa4785a4a8f15156 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "cc8526b37e1a4e17aa4785a4a8f15156" committed_config { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "cc8526b37e1a4e17aa4785a4a8f15156" member_type: VOTER last_known_addr { host: "127.6.234.254" port: 39029 } } peers { permanent_uuid: "4312e033271940b593bea0332e185bff" member_type: VOTER last_known_addr { host: "127.6.234.253" port: 41509 } attrs { promote: false } } peers { permanent_uuid: "ea509afd379942ea887648a5de383571" member_type: NON_VOTER last_known_addr { host: "127.6.234.252" port: 39551 } attrs { promote: true } } }
I20260620 11:42:53.987803 13734 sys_catalog.cc:458] T 00000000000000000000000000000000 P cc8526b37e1a4e17aa4785a4a8f15156 [sys.catalog]: This master's current role is: LEADER
I20260620 11:42:53.988667 13735 raft_consensus.cc:1050] T 00000000000000000000000000000000 P cc8526b37e1a4e17aa4785a4a8f15156: attempting to promote NON_VOTER ea509afd379942ea887648a5de383571 to VOTER
I20260620 11:42:53.988786 13735 consensus_queue.cc:237] T 00000000000000000000000000000000 P cc8526b37e1a4e17aa4785a4a8f15156 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 10, Committed index: 10, Last appended: 1.10, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 11 OBSOLETE_local: true peers { permanent_uuid: "cc8526b37e1a4e17aa4785a4a8f15156" member_type: VOTER last_known_addr { host: "127.6.234.254" port: 39029 } } peers { permanent_uuid: "4312e033271940b593bea0332e185bff" member_type: VOTER last_known_addr { host: "127.6.234.253" port: 41509 } attrs { promote: false } } peers { permanent_uuid: "ea509afd379942ea887648a5de383571" member_type: VOTER last_known_addr { host: "127.6.234.252" port: 39551 } attrs { promote: false } }
I20260620 11:42:53.989189 13783 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 4312e033271940b593bea0332e185bff [term 1 FOLLOWER]: Refusing update from remote peer cc8526b37e1a4e17aa4785a4a8f15156: Log matching property violated. Preceding OpId in replica: term: 1 index: 10. Preceding OpId from leader: term: 1 index: 11. (index mismatch)
I20260620 11:42:53.989212 13976 raft_consensus.cc:1261] T 00000000000000000000000000000000 P ea509afd379942ea887648a5de383571 [term 1 LEARNER]: Refusing update from remote peer cc8526b37e1a4e17aa4785a4a8f15156: Log matching property violated. Preceding OpId in replica: term: 1 index: 10. Preceding OpId from leader: term: 1 index: 11. (index mismatch)
I20260620 11:42:53.989419 13734 consensus_queue.cc:1035] T 00000000000000000000000000000000 P cc8526b37e1a4e17aa4785a4a8f15156 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ea509afd379942ea887648a5de383571" member_type: VOTER last_known_addr { host: "127.6.234.252" port: 39551 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 11, Last known committed idx: 10, Time since last communication: 0.000s
I20260620 11:42:53.989547 13930 consensus_queue.cc:1035] T 00000000000000000000000000000000 P cc8526b37e1a4e17aa4785a4a8f15156 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4312e033271940b593bea0332e185bff" member_type: VOTER last_known_addr { host: "127.6.234.253" port: 41509 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 11, Last known committed idx: 10, Time since last communication: 0.000s
I20260620 11:42:53.990192 13735 raft_consensus.cc:2932] T 00000000000000000000000000000000 P cc8526b37e1a4e17aa4785a4a8f15156 [term 1 LEADER]: Committing config change with OpId 1.11: config changed from index 7 to 11, ea509afd379942ea887648a5de383571 (127.6.234.252) changed from NON_VOTER to VOTER. New config: { opid_index: 11 OBSOLETE_local: true peers { permanent_uuid: "cc8526b37e1a4e17aa4785a4a8f15156" member_type: VOTER last_known_addr { host: "127.6.234.254" port: 39029 } } peers { permanent_uuid: "4312e033271940b593bea0332e185bff" member_type: VOTER last_known_addr { host: "127.6.234.253" port: 41509 } attrs { promote: false } } peers { permanent_uuid: "ea509afd379942ea887648a5de383571" member_type: VOTER last_known_addr { host: "127.6.234.252" port: 39551 } attrs { promote: false } } }
I20260620 11:42:53.990355 13783 raft_consensus.cc:2932] T 00000000000000000000000000000000 P 4312e033271940b593bea0332e185bff [term 1 FOLLOWER]: Committing config change with OpId 1.11: config changed from index 7 to 11, ea509afd379942ea887648a5de383571 (127.6.234.252) changed from NON_VOTER to VOTER. New config: { opid_index: 11 OBSOLETE_local: true peers { permanent_uuid: "cc8526b37e1a4e17aa4785a4a8f15156" member_type: VOTER last_known_addr { host: "127.6.234.254" port: 39029 } } peers { permanent_uuid: "4312e033271940b593bea0332e185bff" member_type: VOTER last_known_addr { host: "127.6.234.253" port: 41509 } attrs { promote: false } } peers { permanent_uuid: "ea509afd379942ea887648a5de383571" member_type: VOTER last_known_addr { host: "127.6.234.252" port: 39551 } attrs { promote: false } } }
I20260620 11:42:53.990384 13976 raft_consensus.cc:2932] T 00000000000000000000000000000000 P ea509afd379942ea887648a5de383571 [term 1 FOLLOWER]: Committing config change with OpId 1.11: config changed from index 7 to 11, ea509afd379942ea887648a5de383571 (127.6.234.252) changed from NON_VOTER to VOTER. New config: { opid_index: 11 OBSOLETE_local: true peers { permanent_uuid: "cc8526b37e1a4e17aa4785a4a8f15156" member_type: VOTER last_known_addr { host: "127.6.234.254" port: 39029 } } peers { permanent_uuid: "4312e033271940b593bea0332e185bff" member_type: VOTER last_known_addr { host: "127.6.234.253" port: 41509 } attrs { promote: false } } peers { permanent_uuid: "ea509afd379942ea887648a5de383571" member_type: VOTER last_known_addr { host: "127.6.234.252" port: 39551 } attrs { promote: false } } }
I20260620 11:42:53.991083 13735 sys_catalog.cc:455] T 00000000000000000000000000000000 P cc8526b37e1a4e17aa4785a4a8f15156 [sys.catalog]: SysCatalogTable state changed. Reason: Config change replication complete. Latest consensus state: current_term: 1 leader_uuid: "cc8526b37e1a4e17aa4785a4a8f15156" committed_config { opid_index: 11 OBSOLETE_local: true peers { permanent_uuid: "cc8526b37e1a4e17aa4785a4a8f15156" member_type: VOTER last_known_addr { host: "127.6.234.254" port: 39029 } } peers { permanent_uuid: "4312e033271940b593bea0332e185bff" member_type: VOTER last_known_addr { host: "127.6.234.253" port: 41509 } attrs { promote: false } } peers { permanent_uuid: "ea509afd379942ea887648a5de383571" member_type: VOTER last_known_addr { host: "127.6.234.252" port: 39551 } attrs { promote: false } } }
I20260620 11:42:53.991184 13735 sys_catalog.cc:458] T 00000000000000000000000000000000 P cc8526b37e1a4e17aa4785a4a8f15156 [sys.catalog]: This master's current role is: LEADER
I20260620 11:42:53.991240 13735 sys_catalog.cc:455] T 00000000000000000000000000000000 P cc8526b37e1a4e17aa4785a4a8f15156 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "cc8526b37e1a4e17aa4785a4a8f15156" committed_config { opid_index: 11 OBSOLETE_local: true peers { permanent_uuid: "cc8526b37e1a4e17aa4785a4a8f15156" member_type: VOTER last_known_addr { host: "127.6.234.254" port: 39029 } } peers { permanent_uuid: "4312e033271940b593bea0332e185bff" member_type: VOTER last_known_addr { host: "127.6.234.253" port: 41509 } attrs { promote: false } } peers { permanent_uuid: "ea509afd379942ea887648a5de383571" member_type: VOTER last_known_addr { host: "127.6.234.252" port: 39551 } attrs { promote: false } } }
I20260620 11:42:53.991210 13930 sys_catalog.cc:455] T 00000000000000000000000000000000 P cc8526b37e1a4e17aa4785a4a8f15156 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "cc8526b37e1a4e17aa4785a4a8f15156" committed_config { opid_index: 11 OBSOLETE_local: true peers { permanent_uuid: "cc8526b37e1a4e17aa4785a4a8f15156" member_type: VOTER last_known_addr { host: "127.6.234.254" port: 39029 } } peers { permanent_uuid: "4312e033271940b593bea0332e185bff" member_type: VOTER last_known_addr { host: "127.6.234.253" port: 41509 } attrs { promote: false } } peers { permanent_uuid: "ea509afd379942ea887648a5de383571" member_type: VOTER last_known_addr { host: "127.6.234.252" port: 39551 } attrs { promote: false } } }
I20260620 11:42:53.991297 13735 sys_catalog.cc:458] T 00000000000000000000000000000000 P cc8526b37e1a4e17aa4785a4a8f15156 [sys.catalog]: This master's current role is: LEADER
I20260620 11:42:53.991310 13930 sys_catalog.cc:458] T 00000000000000000000000000000000 P cc8526b37e1a4e17aa4785a4a8f15156 [sys.catalog]: This master's current role is: LEADER
I20260620 11:42:53.991281 14005 sys_catalog.cc:455] T 00000000000000000000000000000000 P ea509afd379942ea887648a5de383571 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "cc8526b37e1a4e17aa4785a4a8f15156" committed_config { opid_index: 11 OBSOLETE_local: true peers { permanent_uuid: "cc8526b37e1a4e17aa4785a4a8f15156" member_type: VOTER last_known_addr { host: "127.6.234.254" port: 39029 } } peers { permanent_uuid: "4312e033271940b593bea0332e185bff" member_type: VOTER last_known_addr { host: "127.6.234.253" port: 41509 } attrs { promote: false } } peers { permanent_uuid: "ea509afd379942ea887648a5de383571" member_type: VOTER last_known_addr { host: "127.6.234.252" port: 39551 } attrs { promote: false } } }
I20260620 11:42:53.991359 14005 sys_catalog.cc:458] T 00000000000000000000000000000000 P ea509afd379942ea887648a5de383571 [sys.catalog]: This master's current role is: FOLLOWER
I20260620 11:42:53.991590 14460 sys_catalog.cc:455] T 00000000000000000000000000000000 P 4312e033271940b593bea0332e185bff [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "cc8526b37e1a4e17aa4785a4a8f15156" committed_config { opid_index: 11 OBSOLETE_local: true peers { permanent_uuid: "cc8526b37e1a4e17aa4785a4a8f15156" member_type: VOTER last_known_addr { host: "127.6.234.254" port: 39029 } } peers { permanent_uuid: "4312e033271940b593bea0332e185bff" member_type: VOTER last_known_addr { host: "127.6.234.253" port: 41509 } attrs { promote: false } } peers { permanent_uuid: "ea509afd379942ea887648a5de383571" member_type: VOTER last_known_addr { host: "127.6.234.252" port: 39551 } attrs { promote: false } } }
I20260620 11:42:53.991678 14460 sys_catalog.cc:458] T 00000000000000000000000000000000 P 4312e033271940b593bea0332e185bff [sys.catalog]: This master's current role is: FOLLOWER
I20260620 11:42:54.008159 14443 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 11:42:54.006085 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35940 (local address 127.6.234.193:38171)
0620 11:42:54.006125 (+ 40us) server_negotiation.cc:206] Beginning negotiation
0620 11:42:54.006127 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 11:42:54.006136 (+ 9us) server_negotiation.cc:407] Connection header received
0620 11:42:54.006165 (+ 29us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 11:42:54.006166 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 11:42:54.006182 (+ 16us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 11:42:54.006247 (+ 65us) server_negotiation.cc:226] Negotiated authn=TOKEN
0620 11:42:54.006348 (+ 101us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 11:42:54.006860 (+ 512us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 11:42:54.007554 (+ 694us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 11:42:54.007736 (+ 182us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 11:42:54.007794 (+ 58us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 11:42:54.007908 (+ 114us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0620 11:42:54.008003 (+ 95us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0620 11:42:54.008059 (+ 56us) server_negotiation.cc:1035] Waiting for connection context
0620 11:42:54.008090 (+ 31us) server_negotiation.cc:299] Negotiation successful
0620 11:42:54.008106 (+ 16us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":20}
I20260620 11:42:54.009935 14236 raft_consensus.cc:1261] T bbbf5f0125f343c8a6166cde663ebee9 P 553457094e274e55ace2e66d211e3171 [term 1 FOLLOWER]: Refusing update from remote peer 35c3aac7e3ce4bb3bb0294aa36fd45ab: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 11:42:54.009936 14372 raft_consensus.cc:1261] T bbbf5f0125f343c8a6166cde663ebee9 P 212956ba6c57456f9d631b576e69d6a2 [term 1 FOLLOWER]: Refusing update from remote peer 35c3aac7e3ce4bb3bb0294aa36fd45ab: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 11:42:54.010285 14457 consensus_queue.cc:1035] T bbbf5f0125f343c8a6166cde663ebee9 P 35c3aac7e3ce4bb3bb0294aa36fd45ab [LEADER]: Connected to new peer: Peer: permanent_uuid: "553457094e274e55ace2e66d211e3171" member_type: VOTER last_known_addr { host: "127.6.234.194" port: 39263 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 11:42:54.010399 14450 consensus_queue.cc:1035] T bbbf5f0125f343c8a6166cde663ebee9 P 35c3aac7e3ce4bb3bb0294aa36fd45ab [LEADER]: Connected to new peer: Peer: permanent_uuid: "212956ba6c57456f9d631b576e69d6a2" member_type: VOTER last_known_addr { host: "127.6.234.195" port: 39097 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 11:42:54.013447 14464 mvcc.cc:205] Tried to move back new op lower bound from 7298890850341945344 to 7298890850247942144. Current Snapshot: MvccSnapshot[applied={T|T < 7298890850341945344}]
I20260620 11:42:54.014389 14463 mvcc.cc:205] Tried to move back new op lower bound from 7298890850341945344 to 7298890850247942144. Current Snapshot: MvccSnapshot[applied={T|T < 7298890850341945344}]
I20260620 11:42:54.015944 13174 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:51634:
table { table_name: "test-table" } modify_external_catalogs: true
I20260620 11:42:54.016032 13174 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:51634:
table { table_name: "test-table" } modify_external_catalogs: true
I20260620 11:42:54.017274 13174 catalog_manager.cc:5782] T 00000000000000000000000000000000 P cc8526b37e1a4e17aa4785a4a8f15156: Sending DeleteTablet for 3 replicas of tablet bbbf5f0125f343c8a6166cde663ebee9
I20260620 11:42:54.017606 14216 tablet_service.cc:1514] Processing DeleteTablet for tablet bbbf5f0125f343c8a6166cde663ebee9 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 11:42:54 UTC) from {username='oryx', principal='oryx/127.6.234.254@KRBTEST.COM'} at 127.0.0.1:49256
I20260620 11:42:54.017608 14079 tablet_service.cc:1514] Processing DeleteTablet for tablet bbbf5f0125f343c8a6166cde663ebee9 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 11:42:54 UTC) from {username='oryx', principal='oryx/127.6.234.254@KRBTEST.COM'} at 127.0.0.1:35936
I20260620 11:42:54.017846 14473 tablet_replica.cc:330] T bbbf5f0125f343c8a6166cde663ebee9 P 35c3aac7e3ce4bb3bb0294aa36fd45ab: stopping tablet replica
I20260620 11:42:54.017931 14473 raft_consensus.cc:2229] T bbbf5f0125f343c8a6166cde663ebee9 P 35c3aac7e3ce4bb3bb0294aa36fd45ab [term 1 LEADER]: Raft consensus shutting down.
I20260620 11:42:54.018009 14473 raft_consensus.cc:2258] T bbbf5f0125f343c8a6166cde663ebee9 P 35c3aac7e3ce4bb3bb0294aa36fd45ab [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 11:42:54.018000 14352 tablet_service.cc:1514] Processing DeleteTablet for tablet bbbf5f0125f343c8a6166cde663ebee9 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 11:42:54 UTC) from {username='oryx', principal='oryx/127.6.234.254@KRBTEST.COM'} at 127.0.0.1:39722
I20260620 11:42:54.018237 14474 tablet_replica.cc:330] T bbbf5f0125f343c8a6166cde663ebee9 P 212956ba6c57456f9d631b576e69d6a2: stopping tablet replica
I20260620 11:42:54.018285 14472 tablet_replica.cc:330] T bbbf5f0125f343c8a6166cde663ebee9 P 553457094e274e55ace2e66d211e3171: stopping tablet replica
I20260620 11:42:54.018318 14474 raft_consensus.cc:2229] T bbbf5f0125f343c8a6166cde663ebee9 P 212956ba6c57456f9d631b576e69d6a2 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260620 11:42:54.018379 14472 raft_consensus.cc:2229] T bbbf5f0125f343c8a6166cde663ebee9 P 553457094e274e55ace2e66d211e3171 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260620 11:42:54.018406 14474 raft_consensus.cc:2258] T bbbf5f0125f343c8a6166cde663ebee9 P 212956ba6c57456f9d631b576e69d6a2 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 11:42:54.018489 14472 raft_consensus.cc:2258] T bbbf5f0125f343c8a6166cde663ebee9 P 553457094e274e55ace2e66d211e3171 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 11:42:54.018780 14473 ts_tablet_manager.cc:1905] T bbbf5f0125f343c8a6166cde663ebee9 P 35c3aac7e3ce4bb3bb0294aa36fd45ab: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 11:42:54.018913 14472 ts_tablet_manager.cc:1905] T bbbf5f0125f343c8a6166cde663ebee9 P 553457094e274e55ace2e66d211e3171: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 11:42:54.019222 7083 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task1JRg7N/build/release/bin/kudu with pid 13143
I20260620 11:42:54.020476 14473 ts_tablet_manager.cc:1918] T bbbf5f0125f343c8a6166cde663ebee9 P 35c3aac7e3ce4bb3bb0294aa36fd45ab: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260620 11:42:54.020537 14473 log.cc:1196] T bbbf5f0125f343c8a6166cde663ebee9 P 35c3aac7e3ce4bb3bb0294aa36fd45ab: Deleting WAL directory at /tmp/dist-test-task1JRg7N/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781955746769051-7083-0/minicluster-data/ts-0/wal/wals/bbbf5f0125f343c8a6166cde663ebee9
I20260620 11:42:54.020926 14473 ts_tablet_manager.cc:1939] T bbbf5f0125f343c8a6166cde663ebee9 P 35c3aac7e3ce4bb3bb0294aa36fd45ab: Deleting consensus metadata
I20260620 11:42:54.020977 14472 ts_tablet_manager.cc:1918] T bbbf5f0125f343c8a6166cde663ebee9 P 553457094e274e55ace2e66d211e3171: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260620 11:42:54.021041 14472 log.cc:1196] T bbbf5f0125f343c8a6166cde663ebee9 P 553457094e274e55ace2e66d211e3171: Deleting WAL directory at /tmp/dist-test-task1JRg7N/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781955746769051-7083-0/minicluster-data/ts-1/wal/wals/bbbf5f0125f343c8a6166cde663ebee9
I20260620 11:42:54.021291 14472 ts_tablet_manager.cc:1939] T bbbf5f0125f343c8a6166cde663ebee9 P 553457094e274e55ace2e66d211e3171: Deleting consensus metadata
W20260620 11:42:54.025486 14033 connection.cc:535] server connection from 127.0.0.1:35936 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
W20260620 11:42:54.025839 14172 connection.cc:535] server connection from 127.0.0.1:49256 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
I20260620 11:42:54.026034 14474 ts_tablet_manager.cc:1905] T bbbf5f0125f343c8a6166cde663ebee9 P 212956ba6c57456f9d631b576e69d6a2: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 11:42:54.027518 14474 ts_tablet_manager.cc:1918] T bbbf5f0125f343c8a6166cde663ebee9 P 212956ba6c57456f9d631b576e69d6a2: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260620 11:42:54.027593 14474 log.cc:1196] T bbbf5f0125f343c8a6166cde663ebee9 P 212956ba6c57456f9d631b576e69d6a2: Deleting WAL directory at /tmp/dist-test-task1JRg7N/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781955746769051-7083-0/minicluster-data/ts-2/wal/wals/bbbf5f0125f343c8a6166cde663ebee9
I20260620 11:42:54.027844 14474 ts_tablet_manager.cc:1939] T bbbf5f0125f343c8a6166cde663ebee9 P 212956ba6c57456f9d631b576e69d6a2: Deleting consensus metadata
W20260620 11:42:54.028332 14307 connection.cc:441] server connection from 127.0.0.1:39722 torn down before Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:39722 (request call id 1) could send its response
I20260620 11:42:54.036559 13824 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 11:42:54.027370 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38384 (local address 127.6.234.253:41509)
0620 11:42:54.027422 (+ 52us) server_negotiation.cc:206] Beginning negotiation
0620 11:42:54.027424 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 11:42:54.027431 (+ 7us) server_negotiation.cc:407] Connection header received
0620 11:42:54.027475 (+ 44us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 11:42:54.027475 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 11:42:54.027484 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 11:42:54.027550 (+ 66us) server_negotiation.cc:226] Negotiated authn=SASL
0620 11:42:54.028518 (+ 968us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 11:42:54.029286 (+ 768us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 11:42:54.030750 (+ 1464us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 11:42:54.031060 (+ 310us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 11:42:54.031099 (+ 39us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 11:42:54.033023 (+ 1924us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 11:42:54.033024 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 11:42:54.033025 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 11:42:54.033033 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0620 11:42:54.035038 (+ 2005us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 11:42:54.035424 (+ 386us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 11:42:54.035425 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 11:42:54.035426 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 11:42:54.035484 (+ 58us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 11:42:54.035689 (+ 205us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 11:42:54.035690 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 11:42:54.035691 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 11:42:54.036081 (+ 390us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 11:42:54.036153 (+ 72us) server_negotiation.cc:1035] Waiting for connection context
0620 11:42:54.036366 (+ 213us) server_negotiation.cc:299] Negotiation successful
0620 11:42:54.036465 (+ 99us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":22}
I20260620 11:42:54.036765 14152 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 11:42:54.027348 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43436 (local address 127.6.234.252:39551)
0620 11:42:54.027388 (+ 40us) server_negotiation.cc:206] Beginning negotiation
0620 11:42:54.027390 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 11:42:54.027680 (+ 290us) server_negotiation.cc:407] Connection header received
0620 11:42:54.027780 (+ 100us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 11:42:54.027781 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 11:42:54.027791 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 11:42:54.027821 (+ 30us) server_negotiation.cc:226] Negotiated authn=SASL
0620 11:42:54.028557 (+ 736us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 11:42:54.029332 (+ 775us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 11:42:54.031387 (+ 2055us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 11:42:54.032025 (+ 638us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 11:42:54.032069 (+ 44us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 11:42:54.033630 (+ 1561us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 11:42:54.033631 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 11:42:54.033632 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 11:42:54.033638 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0620 11:42:54.035407 (+ 1769us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 11:42:54.035787 (+ 380us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 11:42:54.035789 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 11:42:54.035789 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 11:42:54.035841 (+ 52us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 11:42:54.036074 (+ 233us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 11:42:54.036075 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 11:42:54.036076 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 11:42:54.036208 (+ 132us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 11:42:54.036281 (+ 73us) server_negotiation.cc:1035] Waiting for connection context
0620 11:42:54.036476 (+ 195us) server_negotiation.cc:299] Negotiation successful
0620 11:42:54.036674 (+ 198us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":3,"server-negotiator.queue_time_us":21}
I20260620 11:42:54.037987 14433 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 11:42:54.018920 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43432 (local address 127.6.234.252:39551)
0620 11:42:54.018964 (+ 44us) server_negotiation.cc:206] Beginning negotiation
0620 11:42:54.018966 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 11:42:54.028095 (+ 9129us) server_negotiation.cc:407] Connection header received
0620 11:42:54.028534 (+ 439us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 11:42:54.028535 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 11:42:54.028545 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 11:42:54.028575 (+ 30us) server_negotiation.cc:226] Negotiated authn=SASL
0620 11:42:54.029425 (+ 850us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 11:42:54.030237 (+ 812us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 11:42:54.031338 (+ 1101us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 11:42:54.031603 (+ 265us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 11:42:54.031667 (+ 64us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 11:42:54.034382 (+ 2715us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 11:42:54.034384 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 11:42:54.034384 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 11:42:54.034392 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0620 11:42:54.036568 (+ 2176us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 11:42:54.037144 (+ 576us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 11:42:54.037145 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 11:42:54.037145 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 11:42:54.037193 (+ 48us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 11:42:54.037398 (+ 205us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 11:42:54.037398 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 11:42:54.037399 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 11:42:54.037513 (+ 114us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 11:42:54.037578 (+ 65us) server_negotiation.cc:1035] Waiting for connection context
0620 11:42:54.037818 (+ 240us) server_negotiation.cc:299] Negotiation successful
0620 11:42:54.037895 (+ 77us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":807,"server-negotiator.queue_time_us":23}
I20260620 11:42:55.015246 14482 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 11:42:55.014595 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.234.254:39029 (local address 127.6.234.194:35455)
0620 11:42:55.015125 (+ 530us) negotiation.cc:107] Waiting for socket to connect
0620 11:42:55.015197 (+ 72us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.234.254:39029: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":512,"thread_start_us":61,"threads_started":1}
I20260620 11:42:55.015266 14483 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 11:42:55.014622 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.234.254:39029 (local address 127.6.234.195:34851)
0620 11:42:55.015061 (+ 439us) negotiation.cc:107] Waiting for socket to connect
0620 11:42:55.015178 (+ 117us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.234.254:39029: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":417,"thread_start_us":73,"threads_started":1}
W20260620 11:42:55.015460 14283 heartbeater.cc:644] Failed to heartbeat to 127.6.234.254:39029 (0 consecutive failures): Network error: Failed to send heartbeat to master: Client connection negotiation failed: client connection to 127.6.234.254:39029: connect: Connection refused (error 111)
W20260620 11:42:55.015462 14419 heartbeater.cc:644] Failed to heartbeat to 127.6.234.254:39029 (0 consecutive failures): Network error: Failed to send heartbeat to master: Client connection negotiation failed: client connection to 127.6.234.254:39029: connect: Connection refused (error 111)
I20260620 11:42:55.016049 14484 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 11:42:55.015865 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.234.254:39029 (local address 127.6.234.193:48061)
0620 11:42:55.015962 (+ 97us) negotiation.cc:107] Waiting for socket to connect
0620 11:42:55.016017 (+ 55us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.234.254:39029: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":80,"thread_start_us":42,"threads_started":1}
W20260620 11:42:55.016235 14146 heartbeater.cc:644] Failed to heartbeat to 127.6.234.254:39029 (0 consecutive failures): Network error: Failed to send heartbeat to master: Client connection negotiation failed: client connection to 127.6.234.254:39029: connect: Connection refused (error 111)
I20260620 11:42:55.517436 14485 raft_consensus.cc:482] T 00000000000000000000000000000000 P 4312e033271940b593bea0332e185bff [term 1 FOLLOWER]: Starting pre-election (detected failure of leader cc8526b37e1a4e17aa4785a4a8f15156)
I20260620 11:42:55.517598 14485 raft_consensus.cc:504] T 00000000000000000000000000000000 P 4312e033271940b593bea0332e185bff [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 11 OBSOLETE_local: true peers { permanent_uuid: "cc8526b37e1a4e17aa4785a4a8f15156" member_type: VOTER last_known_addr { host: "127.6.234.254" port: 39029 } } peers { permanent_uuid: "4312e033271940b593bea0332e185bff" member_type: VOTER last_known_addr { host: "127.6.234.253" port: 41509 } attrs { promote: false } } peers { permanent_uuid: "ea509afd379942ea887648a5de383571" member_type: VOTER last_known_addr { host: "127.6.234.252" port: 39551 } attrs { promote: false } }
I20260620 11:42:55.517999 14485 leader_election.cc:288] T 00000000000000000000000000000000 P 4312e033271940b593bea0332e185bff [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers cc8526b37e1a4e17aa4785a4a8f15156 (127.6.234.254:39029), ea509afd379942ea887648a5de383571 (127.6.234.252:39551)
I20260620 11:42:55.518338 14486 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 11:42:55.518096 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.234.254:39029 (local address 127.0.0.1:52094)
0620 11:42:55.518224 (+ 128us) negotiation.cc:107] Waiting for socket to connect
0620 11:42:55.518297 (+ 73us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.234.254:39029: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":112,"thread_start_us":69,"threads_started":1}
W20260620 11:42:55.518478 13758 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.6.234.254:39029: connect: Connection refused (error 111)
I20260620 11:42:55.518513 14488 raft_consensus.cc:482] T 00000000000000000000000000000000 P ea509afd379942ea887648a5de383571 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader cc8526b37e1a4e17aa4785a4a8f15156)
I20260620 11:42:55.518668 14488 raft_consensus.cc:504] T 00000000000000000000000000000000 P ea509afd379942ea887648a5de383571 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 11 OBSOLETE_local: true peers { permanent_uuid: "cc8526b37e1a4e17aa4785a4a8f15156" member_type: VOTER last_known_addr { host: "127.6.234.254" port: 39029 } } peers { permanent_uuid: "4312e033271940b593bea0332e185bff" member_type: VOTER last_known_addr { host: "127.6.234.253" port: 41509 } attrs { promote: false } } peers { permanent_uuid: "ea509afd379942ea887648a5de383571" member_type: VOTER last_known_addr { host: "127.6.234.252" port: 39551 } attrs { promote: false } }
I20260620 11:42:55.518874 14486 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 11:42:55.518787 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.234.254:39029 (local address 127.0.0.1:52106)
0620 11:42:55.518827 (+ 40us) negotiation.cc:107] Waiting for socket to connect
0620 11:42:55.518850 (+ 23us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.234.254:39029: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":27}
I20260620 11:42:55.518931 14488 leader_election.cc:288] T 00000000000000000000000000000000 P ea509afd379942ea887648a5de383571 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers cc8526b37e1a4e17aa4785a4a8f15156 (127.6.234.254:39029), 4312e033271940b593bea0332e185bff (127.6.234.253:41509)
W20260620 11:42:55.518945 13758 leader_election.cc:334] T 00000000000000000000000000000000 P 4312e033271940b593bea0332e185bff [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer cc8526b37e1a4e17aa4785a4a8f15156 (127.6.234.254:39029): Network error: Client connection negotiation failed: client connection to 127.6.234.254:39029: connect: Connection refused (error 111)
I20260620 11:42:55.519171 14491 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 11:42:55.518993 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.234.254:39029 (local address 127.0.0.1:52112)
0620 11:42:55.519096 (+ 103us) negotiation.cc:107] Waiting for socket to connect
0620 11:42:55.519140 (+ 44us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.234.254:39029: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":89,"thread_start_us":71,"threads_started":1}
W20260620 11:42:55.519363 13951 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.6.234.254:39029: connect: Connection refused (error 111)
I20260620 11:42:55.519775 14491 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 11:42:55.519667 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.234.254:39029 (local address 127.0.0.1:52126)
0620 11:42:55.519720 (+ 53us) negotiation.cc:107] Waiting for socket to connect
0620 11:42:55.519751 (+ 31us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.234.254:39029: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":37}
W20260620 11:42:55.519856 13951 leader_election.cc:334] T 00000000000000000000000000000000 P ea509afd379942ea887648a5de383571 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer cc8526b37e1a4e17aa4785a4a8f15156 (127.6.234.254:39029): Network error: Client connection negotiation failed: client connection to 127.6.234.254:39029: connect: Connection refused (error 111)
I20260620 11:42:55.521852 14489 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 11:42:55.518337 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43438 (local address 127.6.234.252:39551)
0620 11:42:55.518450 (+ 113us) server_negotiation.cc:206] Beginning negotiation
0620 11:42:55.518452 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 11:42:55.518463 (+ 11us) server_negotiation.cc:407] Connection header received
0620 11:42:55.518599 (+ 136us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 11:42:55.518600 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 11:42:55.518614 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 11:42:55.518702 (+ 88us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 11:42:55.518913 (+ 211us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 11:42:55.519896 (+ 983us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 11:42:55.520984 (+ 1088us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 11:42:55.521521 (+ 537us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 11:42:55.521581 (+ 60us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 11:42:55.521590 (+ 9us) server_negotiation.cc:1035] Waiting for connection context
0620 11:42:55.521762 (+ 172us) server_negotiation.cc:299] Negotiation successful
0620 11:42:55.521810 (+ 48us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":94,"thread_start_us":64,"threads_started":1}
I20260620 11:42:55.521869 14487 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 11:42:55.518214 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.234.252:39551 (local address 127.0.0.1:43438)
0620 11:42:55.518352 (+ 138us) negotiation.cc:107] Waiting for socket to connect
0620 11:42:55.518365 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0620 11:42:55.518493 (+ 128us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 11:42:55.518723 (+ 230us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 11:42:55.518724 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 11:42:55.518733 (+ 9us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 11:42:55.518833 (+ 100us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 11:42:55.518836 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 11:42:55.519975 (+ 1139us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 11:42:55.519976 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 11:42:55.520900 (+ 924us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 11:42:55.520903 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 11:42:55.521598 (+ 695us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 11:42:55.521599 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 11:42:55.521660 (+ 61us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 11:42:55.521665 (+ 5us) client_negotiation.cc:769] Sending connection context
0620 11:42:55.521759 (+ 94us) client_negotiation.cc:240] Negotiation successful
0620 11:42:55.521801 (+ 42us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":123,"thread_start_us":56,"threads_started":1}
I20260620 11:42:55.522076 13976 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "4312e033271940b593bea0332e185bff" candidate_term: 2 candidate_status { last_received { term: 1 index: 12 } } ignore_live_leader: false dest_uuid: "ea509afd379942ea887648a5de383571" is_pre_election: true
I20260620 11:42:55.522192 13976 raft_consensus.cc:2454] T 00000000000000000000000000000000 P ea509afd379942ea887648a5de383571 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4312e033271940b593bea0332e185bff in term 1.
I20260620 11:42:55.522347 13759 leader_election.cc:302] T 00000000000000000000000000000000 P 4312e033271940b593bea0332e185bff [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 4312e033271940b593bea0332e185bff, ea509afd379942ea887648a5de383571; no voters: cc8526b37e1a4e17aa4785a4a8f15156
I20260620 11:42:55.522431 14492 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 11:42:55.519098 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.234.253:41509 (local address 127.0.0.1:38392)
0620 11:42:55.519268 (+ 170us) negotiation.cc:107] Waiting for socket to connect
0620 11:42:55.519278 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0620 11:42:55.519327 (+ 49us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 11:42:55.519493 (+ 166us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 11:42:55.519494 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 11:42:55.519497 (+ 3us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 11:42:55.519542 (+ 45us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 11:42:55.519544 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 11:42:55.520445 (+ 901us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 11:42:55.520446 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 11:42:55.521414 (+ 968us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 11:42:55.521416 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 11:42:55.522188 (+ 772us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 11:42:55.522190 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 11:42:55.522258 (+ 68us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 11:42:55.522262 (+ 4us) client_negotiation.cc:769] Sending connection context
0620 11:42:55.522317 (+ 55us) client_negotiation.cc:240] Negotiation successful
0620 11:42:55.522360 (+ 43us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":154,"thread_start_us":49,"threads_started":1}
I20260620 11:42:55.522475 14485 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 4312e033271940b593bea0332e185bff [term 1 FOLLOWER]: Leader pre-election won for term 2
I20260620 11:42:55.522507 14485 raft_consensus.cc:482] T 00000000000000000000000000000000 P 4312e033271940b593bea0332e185bff [term 1 FOLLOWER]: Starting leader election (detected failure of leader cc8526b37e1a4e17aa4785a4a8f15156)
I20260620 11:42:55.522532 14485 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 4312e033271940b593bea0332e185bff [term 1 FOLLOWER]: Advancing to term 2
I20260620 11:42:55.522629 14493 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 11:42:55.519143 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38392 (local address 127.6.234.253:41509)
0620 11:42:55.519329 (+ 186us) server_negotiation.cc:206] Beginning negotiation
0620 11:42:55.519331 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 11:42:55.519340 (+ 9us) server_negotiation.cc:407] Connection header received
0620 11:42:55.519366 (+ 26us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 11:42:55.519366 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 11:42:55.519377 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 11:42:55.519446 (+ 69us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 11:42:55.519621 (+ 175us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 11:42:55.520365 (+ 744us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 11:42:55.521500 (+ 1135us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 11:42:55.522092 (+ 592us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 11:42:55.522154 (+ 62us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 11:42:55.522165 (+ 11us) server_negotiation.cc:1035] Waiting for connection context
0620 11:42:55.522525 (+ 360us) server_negotiation.cc:299] Negotiation successful
0620 11:42:55.522572 (+ 47us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":157,"thread_start_us":92,"threads_started":1}
I20260620 11:42:55.522859 13783 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "ea509afd379942ea887648a5de383571" candidate_term: 2 candidate_status { last_received { term: 1 index: 12 } } ignore_live_leader: false dest_uuid: "4312e033271940b593bea0332e185bff" is_pre_election: true
I20260620 11:42:55.523406 14485 raft_consensus.cc:504] T 00000000000000000000000000000000 P 4312e033271940b593bea0332e185bff [term 2 FOLLOWER]: Starting leader election with config: opid_index: 11 OBSOLETE_local: true peers { permanent_uuid: "cc8526b37e1a4e17aa4785a4a8f15156" member_type: VOTER last_known_addr { host: "127.6.234.254" port: 39029 } } peers { permanent_uuid: "4312e033271940b593bea0332e185bff" member_type: VOTER last_known_addr { host: "127.6.234.253" port: 41509 } attrs { promote: false } } peers { permanent_uuid: "ea509afd379942ea887648a5de383571" member_type: VOTER last_known_addr { host: "127.6.234.252" port: 39551 } attrs { promote: false } }
I20260620 11:42:55.523550 14485 leader_election.cc:288] T 00000000000000000000000000000000 P 4312e033271940b593bea0332e185bff [CANDIDATE]: Term 2 election: Requested vote from peers cc8526b37e1a4e17aa4785a4a8f15156 (127.6.234.254:39029), ea509afd379942ea887648a5de383571 (127.6.234.252:39551)
I20260620 11:42:55.523532 13783 raft_consensus.cc:2379] T 00000000000000000000000000000000 P 4312e033271940b593bea0332e185bff [term 2 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate ea509afd379942ea887648a5de383571 in current term 2: Already voted for candidate 4312e033271940b593bea0332e185bff in this term.
I20260620 11:42:55.523707 14487 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 11:42:55.523623 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.234.254:39029 (local address 127.0.0.1:52142)
0620 11:42:55.523661 (+ 38us) negotiation.cc:107] Waiting for socket to connect
0620 11:42:55.523685 (+ 24us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.234.254:39029: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":26}
I20260620 11:42:55.523782 13952 leader_election.cc:302] T 00000000000000000000000000000000 P ea509afd379942ea887648a5de383571 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ea509afd379942ea887648a5de383571; no voters: 4312e033271940b593bea0332e185bff, cc8526b37e1a4e17aa4785a4a8f15156
I20260620 11:42:55.523888 14488 raft_consensus.cc:3037] T 00000000000000000000000000000000 P ea509afd379942ea887648a5de383571 [term 1 FOLLOWER]: Advancing to term 2
I20260620 11:42:55.523973 13976 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "4312e033271940b593bea0332e185bff" candidate_term: 2 candidate_status { last_received { term: 1 index: 12 } } ignore_live_leader: false dest_uuid: "ea509afd379942ea887648a5de383571"
I20260620 11:42:55.524034 14487 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 11:42:55.523918 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.234.254:39029 (local address 127.0.0.1:52144)
0620 11:42:55.523985 (+ 67us) negotiation.cc:107] Waiting for socket to connect
0620 11:42:55.524012 (+ 27us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.234.254:39029: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":50}
W20260620 11:42:55.524113 13758 leader_election.cc:334] T 00000000000000000000000000000000 P 4312e033271940b593bea0332e185bff [CANDIDATE]: Term 2 election: RPC error from VoteRequest() call to peer cc8526b37e1a4e17aa4785a4a8f15156 (127.6.234.254:39029): Network error: Client connection negotiation failed: client connection to 127.6.234.254:39029: connect: Connection refused (error 111)
I20260620 11:42:55.524612 14488 raft_consensus.cc:2726] T 00000000000000000000000000000000 P ea509afd379942ea887648a5de383571 [term 2 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260620 11:42:55.525277 13976 raft_consensus.cc:2454] T 00000000000000000000000000000000 P ea509afd379942ea887648a5de383571 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4312e033271940b593bea0332e185bff in term 2.
I20260620 11:42:55.525552 13759 leader_election.cc:302] T 00000000000000000000000000000000 P 4312e033271940b593bea0332e185bff [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 4312e033271940b593bea0332e185bff, ea509afd379942ea887648a5de383571; no voters: cc8526b37e1a4e17aa4785a4a8f15156
I20260620 11:42:55.525681 14485 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 4312e033271940b593bea0332e185bff [term 2 FOLLOWER]: Leader election won for term 2
I20260620 11:42:55.525894 14485 raft_consensus.cc:686] T 00000000000000000000000000000000 P 4312e033271940b593bea0332e185bff [term 2 LEADER]: Becoming Leader. State: Replica: 4312e033271940b593bea0332e185bff, State: Running, Role: LEADER
I20260620 11:42:55.525981 14485 consensus_queue.cc:237] T 00000000000000000000000000000000 P 4312e033271940b593bea0332e185bff [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 12, Committed index: 12, Last appended: 1.12, Last appended by leader: 12, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 11 OBSOLETE_local: true peers { permanent_uuid: "cc8526b37e1a4e17aa4785a4a8f15156" member_type: VOTER last_known_addr { host: "127.6.234.254" port: 39029 } } peers { permanent_uuid: "4312e033271940b593bea0332e185bff" member_type: VOTER last_known_addr { host: "127.6.234.253" port: 41509 } attrs { promote: false } } peers { permanent_uuid: "ea509afd379942ea887648a5de383571" member_type: VOTER last_known_addr { host: "127.6.234.252" port: 39551 } attrs { promote: false } }
I20260620 11:42:55.526404 14495 sys_catalog.cc:455] T 00000000000000000000000000000000 P 4312e033271940b593bea0332e185bff [sys.catalog]: SysCatalogTable state changed. Reason: New leader 4312e033271940b593bea0332e185bff. Latest consensus state: current_term: 2 leader_uuid: "4312e033271940b593bea0332e185bff" committed_config { opid_index: 11 OBSOLETE_local: true peers { permanent_uuid: "cc8526b37e1a4e17aa4785a4a8f15156" member_type: VOTER last_known_addr { host: "127.6.234.254" port: 39029 } } peers { permanent_uuid: "4312e033271940b593bea0332e185bff" member_type: VOTER last_known_addr { host: "127.6.234.253" port: 41509 } attrs { promote: false } } peers { permanent_uuid: "ea509afd379942ea887648a5de383571" member_type: VOTER last_known_addr { host: "127.6.234.252" port: 39551 } attrs { promote: false } } }
I20260620 11:42:55.526510 14495 sys_catalog.cc:458] T 00000000000000000000000000000000 P 4312e033271940b593bea0332e185bff [sys.catalog]: This master's current role is: LEADER
I20260620 11:42:55.526683 14497 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260620 11:42:55.527139 14497 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260620 11:42:55.527201 14497 catalog_manager.cc:1259] Loaded cluster ID: 62ddcf685f6646508efb7ea8accaf0cf
I20260620 11:42:55.527218 14497 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260620 11:42:55.527412 14497 catalog_manager.cc:1504] Loading token signing keys...
I20260620 11:42:55.527474 14497 catalog_manager.cc:5879] T 00000000000000000000000000000000 P 4312e033271940b593bea0332e185bff: Loaded TSK: 0
I20260620 11:42:55.527565 14497 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260620 11:42:55.570421 13771 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:38384:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260620 11:42:55.572064 13771 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260620 11:42:55.573143 14487 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 11:42:55.573012 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.234.254:39029 (local address 127.0.0.1:52154)
0620 11:42:55.573075 (+ 63us) negotiation.cc:107] Waiting for socket to connect
0620 11:42:55.573114 (+ 39us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.234.254:39029: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":43}
I20260620 11:42:55.573151 13976 raft_consensus.cc:1261] T 00000000000000000000000000000000 P ea509afd379942ea887648a5de383571 [term 2 FOLLOWER]: Refusing update from remote peer 4312e033271940b593bea0332e185bff: Log matching property violated. Preceding OpId in replica: term: 1 index: 12. Preceding OpId from leader: term: 2 index: 14. (index mismatch)
I20260620 11:42:55.573421 14495 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 4312e033271940b593bea0332e185bff [LEADER]: Connected to new peer: Peer: permanent_uuid: "ea509afd379942ea887648a5de383571" member_type: VOTER last_known_addr { host: "127.6.234.252" port: 39551 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 13, Last known committed idx: 12, Time since last communication: 0.000s
I20260620 11:42:55.573601 14487 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 11:42:55.573472 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.234.254:39029 (local address 127.0.0.1:52166)
0620 11:42:55.573539 (+ 67us) negotiation.cc:107] Waiting for socket to connect
0620 11:42:55.573573 (+ 34us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.234.254:39029: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":35}
W20260620 11:42:55.573666 13758 consensus_peers.cc:487] T 00000000000000000000000000000000 P 4312e033271940b593bea0332e185bff -> Peer cc8526b37e1a4e17aa4785a4a8f15156 (127.6.234.254:39029): Couldn't send request to peer cc8526b37e1a4e17aa4785a4a8f15156. Status: Network error: Client connection negotiation failed: client connection to 127.6.234.254:39029: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20260620 11:42:55.574090 14488 sys_catalog.cc:455] T 00000000000000000000000000000000 P ea509afd379942ea887648a5de383571 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 4312e033271940b593bea0332e185bff. Latest consensus state: current_term: 2 leader_uuid: "4312e033271940b593bea0332e185bff" committed_config { opid_index: 11 OBSOLETE_local: true peers { permanent_uuid: "cc8526b37e1a4e17aa4785a4a8f15156" member_type: VOTER last_known_addr { host: "127.6.234.254" port: 39029 } } peers { permanent_uuid: "4312e033271940b593bea0332e185bff" member_type: VOTER last_known_addr { host: "127.6.234.253" port: 41509 } attrs { promote: false } } peers { permanent_uuid: "ea509afd379942ea887648a5de383571" member_type: VOTER last_known_addr { host: "127.6.234.252" port: 39551 } attrs { promote: false } } }
I20260620 11:42:55.574220 14488 sys_catalog.cc:458] T 00000000000000000000000000000000 P ea509afd379942ea887648a5de383571 [sys.catalog]: This master's current role is: FOLLOWER
I20260620 11:42:55.574671 14499 mvcc.cc:205] Tried to move back new op lower bound from 7298890856745390080 to 7298890856555114496. Current Snapshot: MvccSnapshot[applied={T|T < 7298890856745390080}]
I20260620 11:42:55.574698 14485 sys_catalog.cc:455] T 00000000000000000000000000000000 P 4312e033271940b593bea0332e185bff [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 2 leader_uuid: "4312e033271940b593bea0332e185bff" committed_config { opid_index: 11 OBSOLETE_local: true peers { permanent_uuid: "cc8526b37e1a4e17aa4785a4a8f15156" member_type: VOTER last_known_addr { host: "127.6.234.254" port: 39029 } } peers { permanent_uuid: "4312e033271940b593bea0332e185bff" member_type: VOTER last_known_addr { host: "127.6.234.253" port: 41509 } attrs { promote: false } } peers { permanent_uuid: "ea509afd379942ea887648a5de383571" member_type: VOTER last_known_addr { host: "127.6.234.252" port: 39551 } attrs { promote: false } } }
I20260620 11:42:55.574801 14485 sys_catalog.cc:458] T 00000000000000000000000000000000 P 4312e033271940b593bea0332e185bff [sys.catalog]: This master's current role is: LEADER
I20260620 11:42:55.574970 14488 sys_catalog.cc:455] T 00000000000000000000000000000000 P ea509afd379942ea887648a5de383571 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 2 leader_uuid: "4312e033271940b593bea0332e185bff" committed_config { opid_index: 11 OBSOLETE_local: true peers { permanent_uuid: "cc8526b37e1a4e17aa4785a4a8f15156" member_type: VOTER last_known_addr { host: "127.6.234.254" port: 39029 } } peers { permanent_uuid: "4312e033271940b593bea0332e185bff" member_type: VOTER last_known_addr { host: "127.6.234.253" port: 41509 } attrs { promote: false } } peers { permanent_uuid: "ea509afd379942ea887648a5de383571" member_type: VOTER last_known_addr { host: "127.6.234.252" port: 39551 } attrs { promote: false } } }
I20260620 11:42:55.575059 14488 sys_catalog.cc:458] T 00000000000000000000000000000000 P ea509afd379942ea887648a5de383571 [sys.catalog]: This master's current role is: FOLLOWER
I20260620 11:42:55.580035 14487 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 11:42:55.576611 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.234.195:39097 (local address 127.0.0.1:39726)
0620 11:42:55.576652 (+ 41us) negotiation.cc:107] Waiting for socket to connect
0620 11:42:55.576661 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0620 11:42:55.576697 (+ 36us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 11:42:55.576973 (+ 276us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 11:42:55.576975 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 11:42:55.576977 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 11:42:55.577037 (+ 60us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 11:42:55.577039 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 11:42:55.577947 (+ 908us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 11:42:55.577948 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 11:42:55.578994 (+ 1046us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 11:42:55.578996 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 11:42:55.579824 (+ 828us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 11:42:55.579824 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 11:42:55.579883 (+ 59us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 11:42:55.579887 (+ 4us) client_negotiation.cc:769] Sending connection context
0620 11:42:55.579937 (+ 50us) client_negotiation.cc:240] Negotiation successful
0620 11:42:55.579978 (+ 41us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":27}
I20260620 11:42:55.580075 14504 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 11:42:55.576702 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39726 (local address 127.6.234.195:39097)
0620 11:42:55.576837 (+ 135us) server_negotiation.cc:206] Beginning negotiation
0620 11:42:55.576839 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 11:42:55.576847 (+ 8us) server_negotiation.cc:407] Connection header received
0620 11:42:55.576874 (+ 27us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 11:42:55.576875 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 11:42:55.576887 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 11:42:55.576951 (+ 64us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 11:42:55.577128 (+ 177us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 11:42:55.577870 (+ 742us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 11:42:55.579336 (+ 1466us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 11:42:55.579744 (+ 408us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 11:42:55.579826 (+ 82us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 11:42:55.579838 (+ 12us) server_negotiation.cc:1035] Waiting for connection context
0620 11:42:55.579965 (+ 127us) server_negotiation.cc:299] Negotiation successful
0620 11:42:55.580005 (+ 40us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":110,"thread_start_us":78,"threads_started":1}
I20260620 11:42:55.580372 14486 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 11:42:55.576689 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.234.193:38171 (local address 127.0.0.1:35956)
0620 11:42:55.576768 (+ 79us) negotiation.cc:107] Waiting for socket to connect
0620 11:42:55.576776 (+ 8us) client_negotiation.cc:174] Beginning negotiation
0620 11:42:55.576817 (+ 41us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 11:42:55.577183 (+ 366us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 11:42:55.577184 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 11:42:55.577186 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 11:42:55.577223 (+ 37us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 11:42:55.577225 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 11:42:55.578292 (+ 1067us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 11:42:55.578293 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 11:42:55.579341 (+ 1048us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 11:42:55.579343 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 11:42:55.580145 (+ 802us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 11:42:55.580146 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 11:42:55.580204 (+ 58us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 11:42:55.580207 (+ 3us) client_negotiation.cc:769] Sending connection context
0620 11:42:55.580239 (+ 32us) client_negotiation.cc:240] Negotiation successful
0620 11:42:55.580275 (+ 36us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":66,"mutex_wait_us":1}
I20260620 11:42:55.580456 14352 tablet_service.cc:1467] Processing CreateTablet for tablet f6e051e92c014bb1a4c6efc25b228396 (DEFAULT_TABLE table=test-table [id=2b492eb028f2474da181a44e5f655a4d]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 11:42:55.580509 14507 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 11:42:55.576898 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35956 (local address 127.6.234.193:38171)
0620 11:42:55.577023 (+ 125us) server_negotiation.cc:206] Beginning negotiation
0620 11:42:55.577025 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 11:42:55.577033 (+ 8us) server_negotiation.cc:407] Connection header received
0620 11:42:55.577061 (+ 28us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 11:42:55.577062 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 11:42:55.577074 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 11:42:55.577297 (+ 223us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 11:42:55.577342 (+ 45us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 11:42:55.578206 (+ 864us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 11:42:55.579418 (+ 1212us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 11:42:55.580050 (+ 632us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 11:42:55.580350 (+ 300us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 11:42:55.580366 (+ 16us) server_negotiation.cc:1035] Waiting for connection context
0620 11:42:55.580387 (+ 21us) server_negotiation.cc:299] Negotiation successful
0620 11:42:55.580438 (+ 51us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":98,"thread_start_us":65,"threads_started":1}
I20260620 11:42:55.580601 14352 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f6e051e92c014bb1a4c6efc25b228396. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 11:42:55.580598 14506 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 11:42:55.576877 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49268 (local address 127.6.234.194:39263)
0620 11:42:55.577197 (+ 320us) server_negotiation.cc:206] Beginning negotiation
0620 11:42:55.577200 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0620 11:42:55.577219 (+ 19us) server_negotiation.cc:407] Connection header received
0620 11:42:55.577246 (+ 27us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 11:42:55.577248 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 11:42:55.577261 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 11:42:55.577324 (+ 63us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 11:42:55.577481 (+ 157us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 11:42:55.578304 (+ 823us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 11:42:55.579274 (+ 970us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 11:42:55.579906 (+ 632us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 11:42:55.579978 (+ 72us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 11:42:55.579989 (+ 11us) server_negotiation.cc:1035] Waiting for connection context
0620 11:42:55.580464 (+ 475us) server_negotiation.cc:299] Negotiation successful
0620 11:42:55.580515 (+ 51us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":299,"thread_start_us":56,"threads_started":1}
I20260620 11:42:55.580299 14505 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 11:42:55.576703 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.234.194:39263 (local address 127.0.0.1:49268)
0620 11:42:55.577014 (+ 311us) negotiation.cc:107] Waiting for socket to connect
0620 11:42:55.577023 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0620 11:42:55.577066 (+ 43us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 11:42:55.577371 (+ 305us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 11:42:55.577372 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 11:42:55.577374 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 11:42:55.577405 (+ 31us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 11:42:55.577406 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 11:42:55.578377 (+ 971us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 11:42:55.578378 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 11:42:55.579176 (+ 798us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 11:42:55.579178 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 11:42:55.580027 (+ 849us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 11:42:55.580028 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 11:42:55.580100 (+ 72us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 11:42:55.580106 (+ 6us) client_negotiation.cc:769] Sending connection context
0620 11:42:55.580158 (+ 52us) client_negotiation.cc:240] Negotiation successful
0620 11:42:55.580200 (+ 42us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":293,"thread_start_us":78,"threads_started":1}
I20260620 11:42:55.582221 14216 tablet_service.cc:1467] Processing CreateTablet for tablet f6e051e92c014bb1a4c6efc25b228396 (DEFAULT_TABLE table=test-table [id=2b492eb028f2474da181a44e5f655a4d]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 11:42:55.582239 14508 tablet_bootstrap.cc:492] T f6e051e92c014bb1a4c6efc25b228396 P 212956ba6c57456f9d631b576e69d6a2: Bootstrap starting.
I20260620 11:42:55.582214 14079 tablet_service.cc:1467] Processing CreateTablet for tablet f6e051e92c014bb1a4c6efc25b228396 (DEFAULT_TABLE table=test-table [id=2b492eb028f2474da181a44e5f655a4d]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 11:42:55.582346 14079 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f6e051e92c014bb1a4c6efc25b228396. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 11:42:55.582362 14216 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f6e051e92c014bb1a4c6efc25b228396. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 11:42:55.582604 14508 tablet_bootstrap.cc:654] T f6e051e92c014bb1a4c6efc25b228396 P 212956ba6c57456f9d631b576e69d6a2: Neither blocks nor log segments found. Creating new log.
I20260620 11:42:55.583235 14508 tablet_bootstrap.cc:492] T f6e051e92c014bb1a4c6efc25b228396 P 212956ba6c57456f9d631b576e69d6a2: No bootstrap required, opened a new log
I20260620 11:42:55.583292 14508 ts_tablet_manager.cc:1397] T f6e051e92c014bb1a4c6efc25b228396 P 212956ba6c57456f9d631b576e69d6a2: Time spent bootstrapping tablet: real 0.001s user 0.001s sys 0.000s
I20260620 11:42:55.583402 14508 raft_consensus.cc:348] T f6e051e92c014bb1a4c6efc25b228396 P 212956ba6c57456f9d631b576e69d6a2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "212956ba6c57456f9d631b576e69d6a2" member_type: VOTER last_known_addr { host: "127.6.234.195" port: 39097 } } peers { permanent_uuid: "553457094e274e55ace2e66d211e3171" member_type: VOTER last_known_addr { host: "127.6.234.194" port: 39263 } } peers { permanent_uuid: "35c3aac7e3ce4bb3bb0294aa36fd45ab" member_type: VOTER last_known_addr { host: "127.6.234.193" port: 38171 } }
I20260620 11:42:55.583472 14508 raft_consensus.cc:374] T f6e051e92c014bb1a4c6efc25b228396 P 212956ba6c57456f9d631b576e69d6a2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 11:42:55.583491 14508 raft_consensus.cc:729] T f6e051e92c014bb1a4c6efc25b228396 P 212956ba6c57456f9d631b576e69d6a2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 212956ba6c57456f9d631b576e69d6a2, State: Initialized, Role: FOLLOWER
I20260620 11:42:55.583557 14508 consensus_queue.cc:260] T f6e051e92c014bb1a4c6efc25b228396 P 212956ba6c57456f9d631b576e69d6a2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "212956ba6c57456f9d631b576e69d6a2" member_type: VOTER last_known_addr { host: "127.6.234.195" port: 39097 } } peers { permanent_uuid: "553457094e274e55ace2e66d211e3171" member_type: VOTER last_known_addr { host: "127.6.234.194" port: 39263 } } peers { permanent_uuid: "35c3aac7e3ce4bb3bb0294aa36fd45ab" member_type: VOTER last_known_addr { host: "127.6.234.193" port: 38171 } }
I20260620 11:42:55.583717 14508 ts_tablet_manager.cc:1428] T f6e051e92c014bb1a4c6efc25b228396 P 212956ba6c57456f9d631b576e69d6a2: Time spent starting tablet: real 0.000s user 0.001s sys 0.000s
I20260620 11:42:55.583716 14510 tablet_bootstrap.cc:492] T f6e051e92c014bb1a4c6efc25b228396 P 553457094e274e55ace2e66d211e3171: Bootstrap starting.
I20260620 11:42:55.584178 14513 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 11:42:55.583958 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.234.254:39029 (local address 127.6.234.195:42655)
0620 11:42:55.584102 (+ 144us) negotiation.cc:107] Waiting for socket to connect
0620 11:42:55.584137 (+ 35us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.234.254:39029: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":125,"thread_start_us":78,"threads_started":1}
I20260620 11:42:55.584204 14510 tablet_bootstrap.cc:654] T f6e051e92c014bb1a4c6efc25b228396 P 553457094e274e55ace2e66d211e3171: Neither blocks nor log segments found. Creating new log.
I20260620 11:42:55.584414 14420 heartbeater.cc:503] Master 127.6.234.253:41509 was elected leader, sending a full tablet report...
I20260620 11:42:55.584633 14513 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 11:42:55.584398 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.234.254:39029 (local address 127.6.234.195:58541)
0620 11:42:55.584514 (+ 116us) negotiation.cc:107] Waiting for socket to connect
0620 11:42:55.584599 (+ 85us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.234.254:39029: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":105,"thread_start_us":74,"threads_started":1}
I20260620 11:42:55.584780 14510 tablet_bootstrap.cc:492] T f6e051e92c014bb1a4c6efc25b228396 P 553457094e274e55ace2e66d211e3171: No bootstrap required, opened a new log
I20260620 11:42:55.584782 14512 tablet_bootstrap.cc:492] T f6e051e92c014bb1a4c6efc25b228396 P 35c3aac7e3ce4bb3bb0294aa36fd45ab: Bootstrap starting.
I20260620 11:42:55.584832 14510 ts_tablet_manager.cc:1397] T f6e051e92c014bb1a4c6efc25b228396 P 553457094e274e55ace2e66d211e3171: Time spent bootstrapping tablet: real 0.001s user 0.001s sys 0.000s
I20260620 11:42:55.584960 14510 raft_consensus.cc:348] T f6e051e92c014bb1a4c6efc25b228396 P 553457094e274e55ace2e66d211e3171 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "212956ba6c57456f9d631b576e69d6a2" member_type: VOTER last_known_addr { host: "127.6.234.195" port: 39097 } } peers { permanent_uuid: "553457094e274e55ace2e66d211e3171" member_type: VOTER last_known_addr { host: "127.6.234.194" port: 39263 } } peers { permanent_uuid: "35c3aac7e3ce4bb3bb0294aa36fd45ab" member_type: VOTER last_known_addr { host: "127.6.234.193" port: 38171 } }
I20260620 11:42:55.585034 14510 raft_consensus.cc:374] T f6e051e92c014bb1a4c6efc25b228396 P 553457094e274e55ace2e66d211e3171 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 11:42:55.585062 14510 raft_consensus.cc:729] T f6e051e92c014bb1a4c6efc25b228396 P 553457094e274e55ace2e66d211e3171 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 553457094e274e55ace2e66d211e3171, State: Initialized, Role: FOLLOWER
I20260620 11:42:55.585130 14512 tablet_bootstrap.cc:654] T f6e051e92c014bb1a4c6efc25b228396 P 35c3aac7e3ce4bb3bb0294aa36fd45ab: Neither blocks nor log segments found. Creating new log.
I20260620 11:42:55.585109 14510 consensus_queue.cc:260] T f6e051e92c014bb1a4c6efc25b228396 P 553457094e274e55ace2e66d211e3171 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "212956ba6c57456f9d631b576e69d6a2" member_type: VOTER last_known_addr { host: "127.6.234.195" port: 39097 } } peers { permanent_uuid: "553457094e274e55ace2e66d211e3171" member_type: VOTER last_known_addr { host: "127.6.234.194" port: 39263 } } peers { permanent_uuid: "35c3aac7e3ce4bb3bb0294aa36fd45ab" member_type: VOTER last_known_addr { host: "127.6.234.193" port: 38171 } }
I20260620 11:42:55.585280 14510 ts_tablet_manager.cc:1428] T f6e051e92c014bb1a4c6efc25b228396 P 553457094e274e55ace2e66d211e3171: Time spent starting tablet: real 0.000s user 0.001s sys 0.000s
I20260620 11:42:55.585640 14518 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 11:42:55.585445 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.234.254:39029 (local address 127.6.234.194:47875)
0620 11:42:55.585557 (+ 112us) negotiation.cc:107] Waiting for socket to connect
0620 11:42:55.585592 (+ 35us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.234.254:39029: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":93,"thread_start_us":76,"threads_started":1}
I20260620 11:42:55.585879 14512 tablet_bootstrap.cc:492] T f6e051e92c014bb1a4c6efc25b228396 P 35c3aac7e3ce4bb3bb0294aa36fd45ab: No bootstrap required, opened a new log
I20260620 11:42:55.585934 14512 ts_tablet_manager.cc:1397] T f6e051e92c014bb1a4c6efc25b228396 P 35c3aac7e3ce4bb3bb0294aa36fd45ab: Time spent bootstrapping tablet: real 0.001s user 0.001s sys 0.000s
I20260620 11:42:55.586048 14512 raft_consensus.cc:348] T f6e051e92c014bb1a4c6efc25b228396 P 35c3aac7e3ce4bb3bb0294aa36fd45ab [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "212956ba6c57456f9d631b576e69d6a2" member_type: VOTER last_known_addr { host: "127.6.234.195" port: 39097 } } peers { permanent_uuid: "553457094e274e55ace2e66d211e3171" member_type: VOTER last_known_addr { host: "127.6.234.194" port: 39263 } } peers { permanent_uuid: "35c3aac7e3ce4bb3bb0294aa36fd45ab" member_type: VOTER last_known_addr { host: "127.6.234.193" port: 38171 } }
I20260620 11:42:55.586134 14512 raft_consensus.cc:374] T f6e051e92c014bb1a4c6efc25b228396 P 35c3aac7e3ce4bb3bb0294aa36fd45ab [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 11:42:55.586156 14512 raft_consensus.cc:729] T f6e051e92c014bb1a4c6efc25b228396 P 35c3aac7e3ce4bb3bb0294aa36fd45ab [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 35c3aac7e3ce4bb3bb0294aa36fd45ab, State: Initialized, Role: FOLLOWER
I20260620 11:42:55.586215 14512 consensus_queue.cc:260] T f6e051e92c014bb1a4c6efc25b228396 P 35c3aac7e3ce4bb3bb0294aa36fd45ab [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "212956ba6c57456f9d631b576e69d6a2" member_type: VOTER last_known_addr { host: "127.6.234.195" port: 39097 } } peers { permanent_uuid: "553457094e274e55ace2e66d211e3171" member_type: VOTER last_known_addr { host: "127.6.234.194" port: 39263 } } peers { permanent_uuid: "35c3aac7e3ce4bb3bb0294aa36fd45ab" member_type: VOTER last_known_addr { host: "127.6.234.193" port: 38171 } }
I20260620 11:42:55.586372 14512 ts_tablet_manager.cc:1428] T f6e051e92c014bb1a4c6efc25b228396 P 35c3aac7e3ce4bb3bb0294aa36fd45ab: Time spent starting tablet: real 0.000s user 0.001s sys 0.000s
I20260620 11:42:55.586723 14520 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 11:42:55.586515 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.234.254:39029 (local address 127.6.234.193:49873)
0620 11:42:55.586647 (+ 132us) negotiation.cc:107] Waiting for socket to connect
0620 11:42:55.586684 (+ 37us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.234.254:39029: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":106,"thread_start_us":87,"threads_started":1}
I20260620 11:42:55.586804 14147 heartbeater.cc:503] Master 127.6.234.253:41509 was elected leader, sending a full tablet report...
W20260620 11:42:55.586900 14421 tablet.cc:2369] T f6e051e92c014bb1a4c6efc25b228396 P 212956ba6c57456f9d631b576e69d6a2: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260620 11:42:55.587289 14520 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 11:42:55.587215 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.234.254:39029 (local address 127.6.234.193:41267)
0620 11:42:55.587247 (+ 32us) negotiation.cc:107] Waiting for socket to connect
0620 11:42:55.587271 (+ 24us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.234.254:39029: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":14}
I20260620 11:42:55.688647 14519 raft_consensus.cc:482] T f6e051e92c014bb1a4c6efc25b228396 P 35c3aac7e3ce4bb3bb0294aa36fd45ab [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260620 11:42:55.688750 14519 raft_consensus.cc:504] T f6e051e92c014bb1a4c6efc25b228396 P 35c3aac7e3ce4bb3bb0294aa36fd45ab [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "212956ba6c57456f9d631b576e69d6a2" member_type: VOTER last_known_addr { host: "127.6.234.195" port: 39097 } } peers { permanent_uuid: "553457094e274e55ace2e66d211e3171" member_type: VOTER last_known_addr { host: "127.6.234.194" port: 39263 } } peers { permanent_uuid: "35c3aac7e3ce4bb3bb0294aa36fd45ab" member_type: VOTER last_known_addr { host: "127.6.234.193" port: 38171 } }
I20260620 11:42:55.688956 14519 leader_election.cc:288] T f6e051e92c014bb1a4c6efc25b228396 P 35c3aac7e3ce4bb3bb0294aa36fd45ab [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 212956ba6c57456f9d631b576e69d6a2 (127.6.234.195:39097), 553457094e274e55ace2e66d211e3171 (127.6.234.194:39263)
I20260620 11:42:55.689222 14372 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f6e051e92c014bb1a4c6efc25b228396" candidate_uuid: "35c3aac7e3ce4bb3bb0294aa36fd45ab" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "212956ba6c57456f9d631b576e69d6a2" is_pre_election: true
I20260620 11:42:55.689222 14236 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f6e051e92c014bb1a4c6efc25b228396" candidate_uuid: "35c3aac7e3ce4bb3bb0294aa36fd45ab" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "553457094e274e55ace2e66d211e3171" is_pre_election: true
I20260620 11:42:55.689337 14236 raft_consensus.cc:2454] T f6e051e92c014bb1a4c6efc25b228396 P 553457094e274e55ace2e66d211e3171 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 35c3aac7e3ce4bb3bb0294aa36fd45ab in term 0.
I20260620 11:42:55.689337 14372 raft_consensus.cc:2454] T f6e051e92c014bb1a4c6efc25b228396 P 212956ba6c57456f9d631b576e69d6a2 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 35c3aac7e3ce4bb3bb0294aa36fd45ab in term 0.
I20260620 11:42:55.689488 14032 leader_election.cc:302] T f6e051e92c014bb1a4c6efc25b228396 P 35c3aac7e3ce4bb3bb0294aa36fd45ab [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 212956ba6c57456f9d631b576e69d6a2, 35c3aac7e3ce4bb3bb0294aa36fd45ab; no voters:
I20260620 11:42:55.689661 14519 raft_consensus.cc:2781] T f6e051e92c014bb1a4c6efc25b228396 P 35c3aac7e3ce4bb3bb0294aa36fd45ab [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260620 11:42:55.689750 14519 raft_consensus.cc:482] T f6e051e92c014bb1a4c6efc25b228396 P 35c3aac7e3ce4bb3bb0294aa36fd45ab [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260620 11:42:55.689793 14519 raft_consensus.cc:3037] T f6e051e92c014bb1a4c6efc25b228396 P 35c3aac7e3ce4bb3bb0294aa36fd45ab [term 0 FOLLOWER]: Advancing to term 1
I20260620 11:42:55.690515 14519 raft_consensus.cc:504] T f6e051e92c014bb1a4c6efc25b228396 P 35c3aac7e3ce4bb3bb0294aa36fd45ab [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "212956ba6c57456f9d631b576e69d6a2" member_type: VOTER last_known_addr { host: "127.6.234.195" port: 39097 } } peers { permanent_uuid: "553457094e274e55ace2e66d211e3171" member_type: VOTER last_known_addr { host: "127.6.234.194" port: 39263 } } peers { permanent_uuid: "35c3aac7e3ce4bb3bb0294aa36fd45ab" member_type: VOTER last_known_addr { host: "127.6.234.193" port: 38171 } }
I20260620 11:42:55.690654 14519 leader_election.cc:288] T f6e051e92c014bb1a4c6efc25b228396 P 35c3aac7e3ce4bb3bb0294aa36fd45ab [CANDIDATE]: Term 1 election: Requested vote from peers 212956ba6c57456f9d631b576e69d6a2 (127.6.234.195:39097), 553457094e274e55ace2e66d211e3171 (127.6.234.194:39263)
I20260620 11:42:55.690801 14372 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f6e051e92c014bb1a4c6efc25b228396" candidate_uuid: "35c3aac7e3ce4bb3bb0294aa36fd45ab" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "212956ba6c57456f9d631b576e69d6a2"
I20260620 11:42:55.690887 14372 raft_consensus.cc:3037] T f6e051e92c014bb1a4c6efc25b228396 P 212956ba6c57456f9d631b576e69d6a2 [term 0 FOLLOWER]: Advancing to term 1
I20260620 11:42:55.690842 14236 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f6e051e92c014bb1a4c6efc25b228396" candidate_uuid: "35c3aac7e3ce4bb3bb0294aa36fd45ab" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "553457094e274e55ace2e66d211e3171"
I20260620 11:42:55.690932 14236 raft_consensus.cc:3037] T f6e051e92c014bb1a4c6efc25b228396 P 553457094e274e55ace2e66d211e3171 [term 0 FOLLOWER]: Advancing to term 1
I20260620 11:42:55.691535 14372 raft_consensus.cc:2454] T f6e051e92c014bb1a4c6efc25b228396 P 212956ba6c57456f9d631b576e69d6a2 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 35c3aac7e3ce4bb3bb0294aa36fd45ab in term 1.
I20260620 11:42:55.691540 14236 raft_consensus.cc:2454] T f6e051e92c014bb1a4c6efc25b228396 P 553457094e274e55ace2e66d211e3171 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 35c3aac7e3ce4bb3bb0294aa36fd45ab in term 1.
I20260620 11:42:55.691691 14032 leader_election.cc:302] T f6e051e92c014bb1a4c6efc25b228396 P 35c3aac7e3ce4bb3bb0294aa36fd45ab [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 212956ba6c57456f9d631b576e69d6a2, 35c3aac7e3ce4bb3bb0294aa36fd45ab; no voters:
I20260620 11:42:55.691819 14519 raft_consensus.cc:2781] T f6e051e92c014bb1a4c6efc25b228396 P 35c3aac7e3ce4bb3bb0294aa36fd45ab [term 1 FOLLOWER]: Leader election won for term 1
I20260620 11:42:55.691990 14519 raft_consensus.cc:686] T f6e051e92c014bb1a4c6efc25b228396 P 35c3aac7e3ce4bb3bb0294aa36fd45ab [term 1 LEADER]: Becoming Leader. State: Replica: 35c3aac7e3ce4bb3bb0294aa36fd45ab, State: Running, Role: LEADER
I20260620 11:42:55.692068 14519 consensus_queue.cc:237] T f6e051e92c014bb1a4c6efc25b228396 P 35c3aac7e3ce4bb3bb0294aa36fd45ab [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "212956ba6c57456f9d631b576e69d6a2" member_type: VOTER last_known_addr { host: "127.6.234.195" port: 39097 } } peers { permanent_uuid: "553457094e274e55ace2e66d211e3171" member_type: VOTER last_known_addr { host: "127.6.234.194" port: 39263 } } peers { permanent_uuid: "35c3aac7e3ce4bb3bb0294aa36fd45ab" member_type: VOTER last_known_addr { host: "127.6.234.193" port: 38171 } }
I20260620 11:42:55.692358 14520 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 11:42:55.692232 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.234.254:39029 (local address 127.6.234.193:37401)
0620 11:42:55.692270 (+ 38us) negotiation.cc:107] Waiting for socket to connect
0620 11:42:55.692314 (+ 44us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.234.254:39029: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":19}
I20260620 11:42:55.692699 13771 catalog_manager.cc:5495] T f6e051e92c014bb1a4c6efc25b228396 P 35c3aac7e3ce4bb3bb0294aa36fd45ab reported cstate change: term changed from 0 to 1, leader changed from <none> to 35c3aac7e3ce4bb3bb0294aa36fd45ab (127.6.234.193). New cstate: current_term: 1 leader_uuid: "35c3aac7e3ce4bb3bb0294aa36fd45ab" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "212956ba6c57456f9d631b576e69d6a2" member_type: VOTER last_known_addr { host: "127.6.234.195" port: 39097 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "553457094e274e55ace2e66d211e3171" member_type: VOTER last_known_addr { host: "127.6.234.194" port: 39263 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "35c3aac7e3ce4bb3bb0294aa36fd45ab" member_type: VOTER last_known_addr { host: "127.6.234.193" port: 38171 } health_report { overall_health: HEALTHY } } }
I20260620 11:42:55.724957 14507 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 11:42:55.722968 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35972 (local address 127.6.234.193:38171)
0620 11:42:55.723029 (+ 61us) server_negotiation.cc:206] Beginning negotiation
0620 11:42:55.723031 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 11:42:55.723046 (+ 15us) server_negotiation.cc:407] Connection header received
0620 11:42:55.723070 (+ 24us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 11:42:55.723071 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 11:42:55.723080 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 11:42:55.723131 (+ 51us) server_negotiation.cc:226] Negotiated authn=TOKEN
0620 11:42:55.723256 (+ 125us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 11:42:55.723727 (+ 471us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 11:42:55.724412 (+ 685us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 11:42:55.724587 (+ 175us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 11:42:55.724635 (+ 48us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 11:42:55.724759 (+ 124us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0620 11:42:55.724784 (+ 25us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0620 11:42:55.724822 (+ 38us) server_negotiation.cc:1035] Waiting for connection context
0620 11:42:55.724883 (+ 61us) server_negotiation.cc:299] Negotiation successful
0620 11:42:55.724901 (+ 18us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":35}
I20260620 11:42:55.725898 14372 raft_consensus.cc:1261] T f6e051e92c014bb1a4c6efc25b228396 P 212956ba6c57456f9d631b576e69d6a2 [term 1 FOLLOWER]: Refusing update from remote peer 35c3aac7e3ce4bb3bb0294aa36fd45ab: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 11:42:55.725900 14236 raft_consensus.cc:1261] T f6e051e92c014bb1a4c6efc25b228396 P 553457094e274e55ace2e66d211e3171 [term 1 FOLLOWER]: Refusing update from remote peer 35c3aac7e3ce4bb3bb0294aa36fd45ab: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 11:42:55.726122 14521 consensus_queue.cc:1035] T f6e051e92c014bb1a4c6efc25b228396 P 35c3aac7e3ce4bb3bb0294aa36fd45ab [LEADER]: Connected to new peer: Peer: permanent_uuid: "212956ba6c57456f9d631b576e69d6a2" member_type: VOTER last_known_addr { host: "127.6.234.195" port: 39097 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 11:42:55.726234 14519 consensus_queue.cc:1035] T f6e051e92c014bb1a4c6efc25b228396 P 35c3aac7e3ce4bb3bb0294aa36fd45ab [LEADER]: Connected to new peer: Peer: permanent_uuid: "553457094e274e55ace2e66d211e3171" member_type: VOTER last_known_addr { host: "127.6.234.194" port: 39263 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 11:42:55.727133 14514 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 11:42:55.726981 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.234.254:39029 (local address 127.6.234.195:47333)
0620 11:42:55.727044 (+ 63us) negotiation.cc:107] Waiting for socket to connect
0620 11:42:55.727086 (+ 42us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.234.254:39029: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":37}
I20260620 11:42:55.727170 14518 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 11:42:55.727063 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.234.254:39029 (local address 127.6.234.194:59051)
0620 11:42:55.727109 (+ 46us) negotiation.cc:107] Waiting for socket to connect
0620 11:42:55.727138 (+ 29us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.234.254:39029: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":29}
I20260620 11:42:55.727821 14524 mvcc.cc:205] Tried to move back new op lower bound from 7298890857371492352 to 7298890857235255296. Current Snapshot: MvccSnapshot[applied={T|T < 7298890857371492352}]
I20260620 11:42:55.728025 14514 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 11:42:55.727860 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.234.254:39029 (local address 127.6.234.195:58241)
0620 11:42:55.727971 (+ 111us) negotiation.cc:107] Waiting for socket to connect
0620 11:42:55.727999 (+ 28us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.234.254:39029: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":95}
I20260620 11:42:55.728248 14520 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 11:42:55.728112 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.234.254:39029 (local address 127.6.234.193:52253)
0620 11:42:55.728179 (+ 67us) negotiation.cc:107] Waiting for socket to connect
0620 11:42:55.728221 (+ 42us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.234.254:39029: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":51}
I20260620 11:42:55.728241 14284 heartbeater.cc:503] Master 127.6.234.253:41509 was elected leader, sending a full tablet report...
I20260620 11:42:55.729260 13771 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:38384:
table { table_name: "test-table" } modify_external_catalogs: true
I20260620 11:42:55.729353 13771 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:38384:
table { table_name: "test-table" } modify_external_catalogs: true
I20260620 11:42:55.729462 14520 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 11:42:55.729366 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.234.254:39029 (local address 127.6.234.193:36551)
0620 11:42:55.729417 (+ 51us) negotiation.cc:107] Waiting for socket to connect
0620 11:42:55.729441 (+ 24us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.234.254:39029: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":36}
I20260620 11:42:55.730041 14518 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 11:42:55.729893 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.234.254:39029 (local address 127.6.234.194:56679)
0620 11:42:55.729981 (+ 88us) negotiation.cc:107] Waiting for socket to connect
0620 11:42:55.730010 (+ 29us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.234.254:39029: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":60}
I20260620 11:42:55.731053 13771 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 4312e033271940b593bea0332e185bff: Sending DeleteTablet for 3 replicas of tablet f6e051e92c014bb1a4c6efc25b228396
I20260620 11:42:55.731238 14079 tablet_service.cc:1514] Processing DeleteTablet for tablet f6e051e92c014bb1a4c6efc25b228396 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 11:42:55 UTC) from {username='oryx', principal='oryx/127.6.234.253@KRBTEST.COM'} at 127.0.0.1:35956
I20260620 11:42:55.731389 14216 tablet_service.cc:1514] Processing DeleteTablet for tablet f6e051e92c014bb1a4c6efc25b228396 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 11:42:55 UTC) from {username='oryx', principal='oryx/127.6.234.253@KRBTEST.COM'} at 127.0.0.1:49268
I20260620 11:42:55.731424 14533 tablet_replica.cc:330] T f6e051e92c014bb1a4c6efc25b228396 P 35c3aac7e3ce4bb3bb0294aa36fd45ab: stopping tablet replica
I20260620 11:42:55.731472 14533 raft_consensus.cc:2229] T f6e051e92c014bb1a4c6efc25b228396 P 35c3aac7e3ce4bb3bb0294aa36fd45ab [term 1 LEADER]: Raft consensus shutting down.
I20260620 11:42:55.731530 14533 raft_consensus.cc:2258] T f6e051e92c014bb1a4c6efc25b228396 P 35c3aac7e3ce4bb3bb0294aa36fd45ab [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 11:42:55.731554 14352 tablet_service.cc:1514] Processing DeleteTablet for tablet f6e051e92c014bb1a4c6efc25b228396 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 11:42:55 UTC) from {username='oryx', principal='oryx/127.6.234.253@KRBTEST.COM'} at 127.0.0.1:39726
I20260620 11:42:55.731631 7083 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task1JRg7N/build/release/bin/kudu with pid 13920
I20260620 11:42:55.732429 14535 tablet_replica.cc:330] T f6e051e92c014bb1a4c6efc25b228396 P 212956ba6c57456f9d631b576e69d6a2: stopping tablet replica
I20260620 11:42:55.732483 14535 raft_consensus.cc:2229] T f6e051e92c014bb1a4c6efc25b228396 P 212956ba6c57456f9d631b576e69d6a2 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260620 11:42:55.732530 14535 raft_consensus.cc:2258] T f6e051e92c014bb1a4c6efc25b228396 P 212956ba6c57456f9d631b576e69d6a2 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 11:42:55.732743 14534 tablet_replica.cc:330] T f6e051e92c014bb1a4c6efc25b228396 P 553457094e274e55ace2e66d211e3171: stopping tablet replica
I20260620 11:42:55.732820 14534 raft_consensus.cc:2229] T f6e051e92c014bb1a4c6efc25b228396 P 553457094e274e55ace2e66d211e3171 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260620 11:42:55.732869 14534 raft_consensus.cc:2258] T f6e051e92c014bb1a4c6efc25b228396 P 553457094e274e55ace2e66d211e3171 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 11:42:55.733134 14535 ts_tablet_manager.cc:1905] T f6e051e92c014bb1a4c6efc25b228396 P 212956ba6c57456f9d631b576e69d6a2: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 11:42:55.733173 14534 ts_tablet_manager.cc:1905] T f6e051e92c014bb1a4c6efc25b228396 P 553457094e274e55ace2e66d211e3171: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 11:42:55.734846 14535 ts_tablet_manager.cc:1918] T f6e051e92c014bb1a4c6efc25b228396 P 212956ba6c57456f9d631b576e69d6a2: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260620 11:42:55.734891 14535 log.cc:1196] T f6e051e92c014bb1a4c6efc25b228396 P 212956ba6c57456f9d631b576e69d6a2: Deleting WAL directory at /tmp/dist-test-task1JRg7N/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781955746769051-7083-0/minicluster-data/ts-2/wal/wals/f6e051e92c014bb1a4c6efc25b228396
I20260620 11:42:55.734891 14534 ts_tablet_manager.cc:1918] T f6e051e92c014bb1a4c6efc25b228396 P 553457094e274e55ace2e66d211e3171: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260620 11:42:55.734942 14534 log.cc:1196] T f6e051e92c014bb1a4c6efc25b228396 P 553457094e274e55ace2e66d211e3171: Deleting WAL directory at /tmp/dist-test-task1JRg7N/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781955746769051-7083-0/minicluster-data/ts-1/wal/wals/f6e051e92c014bb1a4c6efc25b228396
I20260620 11:42:55.735052 14535 ts_tablet_manager.cc:1939] T f6e051e92c014bb1a4c6efc25b228396 P 212956ba6c57456f9d631b576e69d6a2: Deleting consensus metadata
I20260620 11:42:55.735172 14534 ts_tablet_manager.cc:1939] T f6e051e92c014bb1a4c6efc25b228396 P 553457094e274e55ace2e66d211e3171: Deleting consensus metadata
I20260620 11:42:55.735333 13757 catalog_manager.cc:4841] TS 212956ba6c57456f9d631b576e69d6a2 (127.6.234.195:39097): tablet f6e051e92c014bb1a4c6efc25b228396 (table test-table [id=2b492eb028f2474da181a44e5f655a4d]) successfully deleted
I20260620 11:42:55.735489 13760 catalog_manager.cc:4841] TS 553457094e274e55ace2e66d211e3171 (127.6.234.194:39263): tablet f6e051e92c014bb1a4c6efc25b228396 (table test-table [id=2b492eb028f2474da181a44e5f655a4d]) successfully deleted
W20260620 11:42:55.738274 13758 connection.cc:535] client connection to 127.6.234.193:38171 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
W20260620 11:42:55.738647 14505 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 11:42:55.738521 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.234.193:38171 (local address 127.0.0.1:35988)
0620 11:42:55.738584 (+ 63us) negotiation.cc:107] Waiting for socket to connect
0620 11:42:55.738621 (+ 37us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.234.193:38171: connect: Connection reset by peer (error 104)
Metrics: {"client-negotiator.queue_time_us":43}
I20260620 11:42:55.738801 7083 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task1JRg7N/build/release/bin/kudu with pid 14153
W20260620 11:42:55.738864 13758 catalog_manager.cc:4584] TS 35c3aac7e3ce4bb3bb0294aa36fd45ab (127.6.234.193:38171): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet f6e051e92c014bb1a4c6efc25b228396: Network error: Client connection negotiation failed: client connection to 127.6.234.193:38171: connect: Connection reset by peer (error 104)
I20260620 11:42:55.743635 7083 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task1JRg7N/build/release/bin/kudu with pid 14289
I20260620 11:42:55.749022 7083 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task1JRg7N/build/release/bin/kudu with pid 13639
I20260620 11:42:55.753767 7083 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task1JRg7N/build/release/bin/kudu with pid 13827
2026-06-20T11:42:55Z chronyd exiting
[ OK ] SecurityITest.TestNonDefaultPrincipalMultipleMaster (4637 ms)
[ RUN ] SecurityITest.TestNonDefaultPrincipal
Loading random data
Initializing database '/tmp/dist-test-task1JRg7N/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781955746769051-7083-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 20 11:42:55 dist-test-slave-cjc6 krb5kdc[14538](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 20 11:42:55 dist-test-slave-cjc6 krb5kdc[14538](info): set up 2 sockets
Jun 20 11:42:55 dist-test-slave-cjc6 krb5kdc[14538](info): commencing operation
krb5kdc: starting...
W20260620 11:42:56.794045 7083 mini_kdc.cc:121] Time spent starting KDC: real 1.018s user 0.000s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task1JRg7N/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781955746769051-7083-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task1JRg7N/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781955746769051-7083-0/krb5kdc/test-user.keytab.
Jun 20 11:42:56 dist-test-slave-cjc6 krb5kdc[14538](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781955776, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-20T11:42:56Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-20T11:42:56Z Disabled control of system clock
WARNING: no policy specified for oryx/127.6.234.254@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.6.234.254@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.6.234.254 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task1JRg7N/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781955746769051-7083-0/krb5kdc/oryx_127.6.234.254.keytab.
Entry for principal oryx/127.6.234.254 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task1JRg7N/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781955746769051-7083-0/krb5kdc/oryx_127.6.234.254.keytab.
I20260620 11:42:56.906554 7083 external_mini_cluster.cc:1326] Running /tmp/dist-test-task1JRg7N/build/release/bin/kudu
/tmp/dist-test-task1JRg7N/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task1JRg7N/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781955746769051-7083-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task1JRg7N/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781955746769051-7083-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task1JRg7N/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781955746769051-7083-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task1JRg7N/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781955746769051-7083-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.6.234.254:44159
--webserver_interface=127.6.234.254
--webserver_port=0
--keytab_file=/tmp/dist-test-task1JRg7N/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781955746769051-7083-0/krb5kdc/oryx_127.6.234.254.keytab
--principal=oryx/127.6.234.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.6.234.212:41127
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.6.234.254:44159
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--txn_manager_enabled=true with env {KRB5CCNAME=/tmp/dist-test-task1JRg7N/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781955746769051-7083-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task1JRg7N/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781955746769051-7083-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task1JRg7N/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781955746769051-7083-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 11:42:56.985647 14551 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 11:42:56.985952 14551 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 11:42:56.985975 14551 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 11:42:56.987138 14551 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 11:42:56.987178 14551 flags.cc:425] Enabled experimental flag: --txn_manager_enabled=true
W20260620 11:42:56.987192 14551 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 11:42:56.987236 14551 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 11:42:56.987267 14551 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 11:42:56.987284 14551 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 11:42:56.988806 14551 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.6.234.212:41127
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task1JRg7N/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781955746769051-7083-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task1JRg7N/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781955746769051-7083-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.6.234.254:44159
--txn_manager_enabled=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task1JRg7N/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781955746769051-7083-0/krb5kdc/oryx_127.6.234.254.keytab
--principal=oryx/127.6.234.254
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.6.234.254:44159
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task1JRg7N/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781955746769051-7083-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.6.234.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.14551
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task1JRg7N/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781955746769051-7083-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 20 Jun 2026 10:32:37 UTC on ccycloud.quasar-bgyofv.root.comops.site
build id 4457173
I20260620 11:42:56.989068 14551 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 11:42:56.989303 14551 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 11:42:56.994007 14559 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 11:42:56.994040 14556 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 11:42:56.994225 14557 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 11:42:56.994244 14551 server_base.cc:1034] running on GCE node
I20260620 11:42:56.994465 14551 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 11:42:56.994740 14551 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 11:42:56.995910 14551 hybrid_clock.cc:648] HybridClock initialized: now 1781955776995920 us; error 54 us; skew 500 ppm
Jun 20 11:42:56 dist-test-slave-cjc6 krb5kdc[14538](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781955776, etypes {rep=17 tkt=17 ses=17}, oryx/127.6.234.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 11:42:56.998528 14551 init.cc:377] Logged in from keytab as oryx/127.6.234.254@KRBTEST.COM (short username oryx)
I20260620 11:42:56.998986 14551 webserver.cc:460] Webserver started at http://127.6.234.254:33757/ using document root <none> and password file <none>
I20260620 11:42:56.999219 14551 fs_manager.cc:359] Metadata directory not provided
I20260620 11:42:56.999265 14551 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 11:42:56.999408 14551 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 11:42:57.000442 14551 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task1JRg7N/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781955746769051-7083-0/minicluster-data/master-0/data/instance:
uuid: "f75e5ca955994ed9ba7f8b6ce11b9c5a"
format_stamp: "Formatted at 2026-06-20 11:42:56 on dist-test-slave-cjc6"
server_key: "e8247c220f52ddc09108e97d3d4fac4d"
server_key_iv: "39f6e9fdd8a03e6a921eb16df41b8954"
server_key_version: "encryptionkey@0"
I20260620 11:42:57.000799 14551 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task1JRg7N/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781955746769051-7083-0/minicluster-data/master-0/wal/instance:
uuid: "f75e5ca955994ed9ba7f8b6ce11b9c5a"
format_stamp: "Formatted at 2026-06-20 11:42:56 on dist-test-slave-cjc6"
server_key: "e8247c220f52ddc09108e97d3d4fac4d"
server_key_iv: "39f6e9fdd8a03e6a921eb16df41b8954"
server_key_version: "encryptionkey@0"
I20260620 11:42:57.002135 14551 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260620 11:42:57.002853 14566 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 11:42:57.003036 14551 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260620 11:42:57.003130 14551 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task1JRg7N/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781955746769051-7083-0/minicluster-data/master-0/data,/tmp/dist-test-task1JRg7N/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781955746769051-7083-0/minicluster-data/master-0/wal
uuid: "f75e5ca955994ed9ba7f8b6ce11b9c5a"
format_stamp: "Formatted at 2026-06-20 11:42:56 on dist-test-slave-cjc6"
server_key: "e8247c220f52ddc09108e97d3d4fac4d"
server_key_iv: "39f6e9fdd8a03e6a921eb16df41b8954"
server_key_version: "encryptionkey@0"
I20260620 11:42:57.003206 14551 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task1JRg7N/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781955746769051-7083-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task1JRg7N/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781955746769051-7083-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task1JRg7N/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781955746769051-7083-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 11:42:57.036221 14551 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 11:42:57.038817 14551 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 11:42:57.038980 14551 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 11:42:57.043776 14551 rpc_server.cc:307] RPC server started. Bound to: 127.6.234.254:44159
I20260620 11:42:57.043830 14628 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.6.234.254:44159 every 8 connection(s)
I20260620 11:42:57.044155 14551 server_base.cc:1166] Dumped server information to /tmp/dist-test-task1JRg7N/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781955746769051-7083-0/minicluster-data/master-0/data/info.pb
I20260620 11:42:57.044755 14629 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 11:42:57.046988 14629 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P f75e5ca955994ed9ba7f8b6ce11b9c5a: Bootstrap starting.
I20260620 11:42:57.047585 14629 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P f75e5ca955994ed9ba7f8b6ce11b9c5a: Neither blocks nor log segments found. Creating new log.
I20260620 11:42:57.047878 14629 log.cc:824] T 00000000000000000000000000000000 P f75e5ca955994ed9ba7f8b6ce11b9c5a: Log is configured to *not* fsync() on all Append() calls
I20260620 11:42:57.048599 14629 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P f75e5ca955994ed9ba7f8b6ce11b9c5a: No bootstrap required, opened a new log
I20260620 11:42:57.050665 14629 raft_consensus.cc:348] T 00000000000000000000000000000000 P f75e5ca955994ed9ba7f8b6ce11b9c5a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f75e5ca955994ed9ba7f8b6ce11b9c5a" member_type: VOTER last_known_addr { host: "127.6.234.254" port: 44159 } }
I20260620 11:42:57.050844 14629 raft_consensus.cc:374] T 00000000000000000000000000000000 P f75e5ca955994ed9ba7f8b6ce11b9c5a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 11:42:57.050896 14629 raft_consensus.cc:729] T 00000000000000000000000000000000 P f75e5ca955994ed9ba7f8b6ce11b9c5a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f75e5ca955994ed9ba7f8b6ce11b9c5a, State: Initialized, Role: FOLLOWER
I20260620 11:42:57.050931 7083 external_mini_cluster.cc:1388] Started /tmp/dist-test-task1JRg7N/build/release/bin/kudu as pid 14551
I20260620 11:42:57.051054 7083 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task1JRg7N/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781955746769051-7083-0/minicluster-data/master-0/wal/instance
I20260620 11:42:57.051074 14629 consensus_queue.cc:260] T 00000000000000000000000000000000 P f75e5ca955994ed9ba7f8b6ce11b9c5a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f75e5ca955994ed9ba7f8b6ce11b9c5a" member_type: VOTER last_known_addr { host: "127.6.234.254" port: 44159 } }
I20260620 11:42:57.051177 14629 raft_consensus.cc:388] T 00000000000000000000000000000000 P f75e5ca955994ed9ba7f8b6ce11b9c5a [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260620 11:42:57.051218 14629 raft_consensus.cc:482] T 00000000000000000000000000000000 P f75e5ca955994ed9ba7f8b6ce11b9c5a [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260620 11:42:57.051275 7083 external_mini_cluster.cc:1428] Setting key c20e56082578f7eabb22c35717658667
I20260620 11:42:57.051275 14629 raft_consensus.cc:3037] T 00000000000000000000000000000000 P f75e5ca955994ed9ba7f8b6ce11b9c5a [term 0 FOLLOWER]: Advancing to term 1
I20260620 11:42:57.051803 14629 raft_consensus.cc:504] T 00000000000000000000000000000000 P f75e5ca955994ed9ba7f8b6ce11b9c5a [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f75e5ca955994ed9ba7f8b6ce11b9c5a" member_type: VOTER last_known_addr { host: "127.6.234.254" port: 44159 } }
I20260620 11:42:57.051911 14629 leader_election.cc:302] T 00000000000000000000000000000000 P f75e5ca955994ed9ba7f8b6ce11b9c5a [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: f75e5ca955994ed9ba7f8b6ce11b9c5a; no voters:
I20260620 11:42:57.052052 14629 leader_election.cc:288] T 00000000000000000000000000000000 P f75e5ca955994ed9ba7f8b6ce11b9c5a [CANDIDATE]: Term 1 election: Requested vote from peers
I20260620 11:42:57.052083 14634 raft_consensus.cc:2781] T 00000000000000000000000000000000 P f75e5ca955994ed9ba7f8b6ce11b9c5a [term 1 FOLLOWER]: Leader election won for term 1
I20260620 11:42:57.052181 14634 raft_consensus.cc:686] T 00000000000000000000000000000000 P f75e5ca955994ed9ba7f8b6ce11b9c5a [term 1 LEADER]: Becoming Leader. State: Replica: f75e5ca955994ed9ba7f8b6ce11b9c5a, State: Running, Role: LEADER
I20260620 11:42:57.052260 14634 consensus_queue.cc:237] T 00000000000000000000000000000000 P f75e5ca955994ed9ba7f8b6ce11b9c5a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f75e5ca955994ed9ba7f8b6ce11b9c5a" member_type: VOTER last_known_addr { host: "127.6.234.254" port: 44159 } }
I20260620 11:42:57.052307 14629 sys_catalog.cc:564] T 00000000000000000000000000000000 P f75e5ca955994ed9ba7f8b6ce11b9c5a [sys.catalog]: configured and running, proceeding with master startup.
I20260620 11:42:57.052579 14636 sys_catalog.cc:455] T 00000000000000000000000000000000 P f75e5ca955994ed9ba7f8b6ce11b9c5a [sys.catalog]: SysCatalogTable state changed. Reason: New leader f75e5ca955994ed9ba7f8b6ce11b9c5a. Latest consensus state: current_term: 1 leader_uuid: "f75e5ca955994ed9ba7f8b6ce11b9c5a" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f75e5ca955994ed9ba7f8b6ce11b9c5a" member_type: VOTER last_known_addr { host: "127.6.234.254" port: 44159 } } }
I20260620 11:42:57.052671 14636 sys_catalog.cc:458] T 00000000000000000000000000000000 P f75e5ca955994ed9ba7f8b6ce11b9c5a [sys.catalog]: This master's current role is: LEADER
I20260620 11:42:57.052837 14635 sys_catalog.cc:455] T 00000000000000000000000000000000 P f75e5ca955994ed9ba7f8b6ce11b9c5a [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "f75e5ca955994ed9ba7f8b6ce11b9c5a" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f75e5ca955994ed9ba7f8b6ce11b9c5a" member_type: VOTER last_known_addr { host: "127.6.234.254" port: 44159 } } }
I20260620 11:42:57.052915 14635 sys_catalog.cc:458] T 00000000000000000000000000000000 P f75e5ca955994ed9ba7f8b6ce11b9c5a [sys.catalog]: This master's current role is: LEADER
I20260620 11:42:57.053033 14640 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260620 11:42:57.053522 14640 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260620 11:42:57.055696 14640 catalog_manager.cc:1347] Generated new cluster ID: c2241e0c146a42c1b459d051a82c6509
I20260620 11:42:57.055747 14640 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
Jun 20 11:42:57 dist-test-slave-cjc6 krb5kdc[14538](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781955776, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.6.234.254@KRBTEST.COM
I20260620 11:42:57.062994 14633 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 11:42:57.051827 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46530 (local address 127.6.234.254:44159)
0620 11:42:57.052009 (+ 182us) server_negotiation.cc:206] Beginning negotiation
0620 11:42:57.052019 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0620 11:42:57.052050 (+ 31us) server_negotiation.cc:407] Connection header received
0620 11:42:57.052185 (+ 135us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 11:42:57.052192 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 11:42:57.052258 (+ 66us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 11:42:57.053233 (+ 975us) server_negotiation.cc:226] Negotiated authn=SASL
0620 11:42:57.053342 (+ 109us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 11:42:57.054564 (+ 1222us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 11:42:57.055572 (+ 1008us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 11:42:57.056301 (+ 729us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 11:42:57.056410 (+ 109us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 11:42:57.059278 (+ 2868us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 11:42:57.059282 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 11:42:57.059282 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 11:42:57.059293 (+ 11us) server_negotiation.cc:922] calling sasl_server_start()
0620 11:42:57.061269 (+ 1976us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 11:42:57.061675 (+ 406us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 11:42:57.061678 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 11:42:57.061678 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 11:42:57.061812 (+ 134us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 11:42:57.062032 (+ 220us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 11:42:57.062033 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 11:42:57.062033 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 11:42:57.062317 (+ 284us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 11:42:57.062472 (+ 155us) server_negotiation.cc:1035] Waiting for connection context
0620 11:42:57.062582 (+ 110us) server_negotiation.cc:299] Negotiation successful
0620 11:42:57.062701 (+ 119us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":131,"thread_start_us":75,"threads_started":1}
I20260620 11:42:57.064863 14640 catalog_manager.cc:1370] Generated new certificate authority record
I20260620 11:42:57.065358 14640 catalog_manager.cc:1504] Loading token signing keys...
I20260620 11:42:57.068029 14640 catalog_manager.cc:5868] T 00000000000000000000000000000000 P f75e5ca955994ed9ba7f8b6ce11b9c5a: Generated new TSK 0
I20260620 11:42:57.068176 14640 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for oryx/127.6.234.193@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.6.234.193@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.6.234.193 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task1JRg7N/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781955746769051-7083-0/krb5kdc/oryx_127.6.234.193.keytab.
Entry for principal oryx/127.6.234.193 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task1JRg7N/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781955746769051-7083-0/krb5kdc/oryx_127.6.234.193.keytab.
I20260620 11:42:57.096714 7083 external_mini_cluster.cc:1326] Running /tmp/dist-test-task1JRg7N/build/release/bin/kudu
/tmp/dist-test-task1JRg7N/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task1JRg7N/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781955746769051-7083-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-task1JRg7N/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781955746769051-7083-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task1JRg7N/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781955746769051-7083-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-task1JRg7N/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781955746769051-7083-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.6.234.193:0
--local_ip_for_outbound_sockets=127.6.234.193
--webserver_interface=127.6.234.193
--webserver_port=0
--tserver_master_addrs=127.6.234.254:44159
--keytab_file=/tmp/dist-test-task1JRg7N/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781955746769051-7083-0/krb5kdc/oryx_127.6.234.193.keytab
--principal=oryx/127.6.234.193
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.6.234.212:41127
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true with env {KRB5CCNAME=/tmp/dist-test-task1JRg7N/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781955746769051-7083-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task1JRg7N/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781955746769051-7083-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task1JRg7N/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781955746769051-7083-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 11:42:57.182071 14655 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 11:42:57.182292 14655 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 11:42:57.182330 14655 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 11:42:57.183455 14655 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 11:42:57.183518 14655 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 11:42:57.183552 14655 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260620 11:42:57.183593 14655 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.6.234.193
I20260620 11:42:57.185081 14655 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.6.234.212:41127
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task1JRg7N/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781955746769051-7083-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-task1JRg7N/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781955746769051-7083-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task1JRg7N/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781955746769051-7083-0/krb5kdc/oryx_127.6.234.193.keytab
--principal=oryx/127.6.234.193
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.6.234.193:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task1JRg7N/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781955746769051-7083-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.6.234.193
--webserver_port=0
--enable_txn_system_client_init=true
--tserver_master_addrs=127.6.234.254:44159
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.14655
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.6.234.193
--log_dir=/tmp/dist-test-task1JRg7N/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781955746769051-7083-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 20 Jun 2026 10:32:37 UTC on ccycloud.quasar-bgyofv.root.comops.site
build id 4457173
I20260620 11:42:57.185333 14655 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 11:42:57.185570 14655 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 11:42:57.190474 14660 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 11:42:57.190454 14661 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 11:42:57.190647 14655 server_base.cc:1034] running on GCE node
W20260620 11:42:57.190703 14663 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 11:42:57.190914 14655 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 11:42:57.191154 14655 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 11:42:57.192313 14655 hybrid_clock.cc:648] HybridClock initialized: now 1781955777192316 us; error 38 us; skew 500 ppm
Jun 20 11:42:57 dist-test-slave-cjc6 krb5kdc[14538](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781955777, etypes {rep=17 tkt=17 ses=17}, oryx/127.6.234.193@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 11:42:57.194715 14655 init.cc:377] Logged in from keytab as oryx/127.6.234.193@KRBTEST.COM (short username oryx)
I20260620 11:42:57.195132 14655 webserver.cc:460] Webserver started at http://127.6.234.193:40693/ using document root <none> and password file <none>
I20260620 11:42:57.195353 14655 fs_manager.cc:359] Metadata directory not provided
I20260620 11:42:57.195418 14655 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 11:42:57.195565 14655 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 11:42:57.196589 14655 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task1JRg7N/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781955746769051-7083-0/minicluster-data/ts-0/data/instance:
uuid: "580c848b417d4ffbab6d865896c13de7"
format_stamp: "Formatted at 2026-06-20 11:42:57 on dist-test-slave-cjc6"
server_key: "ff60cfaf1db1089de1e7c13f7716276b"
server_key_iv: "8c650b1f9704465a5552c61a6895e017"
server_key_version: "encryptionkey@0"
I20260620 11:42:57.196936 14655 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task1JRg7N/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781955746769051-7083-0/minicluster-data/ts-0/wal/instance:
uuid: "580c848b417d4ffbab6d865896c13de7"
format_stamp: "Formatted at 2026-06-20 11:42:57 on dist-test-slave-cjc6"
server_key: "ff60cfaf1db1089de1e7c13f7716276b"
server_key_iv: "8c650b1f9704465a5552c61a6895e017"
server_key_version: "encryptionkey@0"
I20260620 11:42:57.198297 14655 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260620 11:42:57.198990 14670 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 11:42:57.199149 14655 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260620 11:42:57.199246 14655 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task1JRg7N/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781955746769051-7083-0/minicluster-data/ts-0/data,/tmp/dist-test-task1JRg7N/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781955746769051-7083-0/minicluster-data/ts-0/wal
uuid: "580c848b417d4ffbab6d865896c13de7"
format_stamp: "Formatted at 2026-06-20 11:42:57 on dist-test-slave-cjc6"
server_key: "ff60cfaf1db1089de1e7c13f7716276b"
server_key_iv: "8c650b1f9704465a5552c61a6895e017"
server_key_version: "encryptionkey@0"
I20260620 11:42:57.199342 14655 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task1JRg7N/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781955746769051-7083-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-task1JRg7N/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781955746769051-7083-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-task1JRg7N/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781955746769051-7083-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 11:42:57.209841 14655 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 11:42:57.211639 14655 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 11:42:57.211804 14655 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 11:42:57.212268 14655 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 11:42:57.212343 14655 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 11:42:57.212406 14655 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 11:42:57.212438 14655 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 11:42:57.219166 14655 rpc_server.cc:307] RPC server started. Bound to: 127.6.234.193:44461
I20260620 11:42:57.219192 14785 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.6.234.193:44461 every 8 connection(s)
Jun 20 11:42:57 dist-test-slave-cjc6 krb5kdc[14538](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781955777, etypes {rep=17 tkt=17 ses=17}, oryx/127.6.234.193@KRBTEST.COM for oryx/127.6.234.254@KRBTEST.COM
I20260620 11:42:57.219553 14655 server_base.cc:1166] Dumped server information to /tmp/dist-test-task1JRg7N/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781955746769051-7083-0/minicluster-data/ts-0/data/info.pb
I20260620 11:42:57.222568 14633 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 11:42:57.212721 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.6.234.193:50381 (local address 127.6.234.254:44159)
0620 11:42:57.212795 (+ 74us) server_negotiation.cc:206] Beginning negotiation
0620 11:42:57.212798 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0620 11:42:57.212985 (+ 187us) server_negotiation.cc:407] Connection header received
0620 11:42:57.213458 (+ 473us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 11:42:57.213460 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 11:42:57.213476 (+ 16us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 11:42:57.213557 (+ 81us) server_negotiation.cc:226] Negotiated authn=SASL
0620 11:42:57.214356 (+ 799us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 11:42:57.215219 (+ 863us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 11:42:57.216782 (+ 1563us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 11:42:57.217162 (+ 380us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 11:42:57.217248 (+ 86us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 11:42:57.219926 (+ 2678us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 11:42:57.219928 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 11:42:57.219928 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 11:42:57.219940 (+ 12us) server_negotiation.cc:922] calling sasl_server_start()
0620 11:42:57.221418 (+ 1478us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 11:42:57.221805 (+ 387us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 11:42:57.221806 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 11:42:57.221806 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 11:42:57.221872 (+ 66us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 11:42:57.222083 (+ 211us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 11:42:57.222084 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 11:42:57.222084 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 11:42:57.222181 (+ 97us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 11:42:57.222226 (+ 45us) server_negotiation.cc:1035] Waiting for connection context
0620 11:42:57.222445 (+ 219us) server_negotiation.cc:299] Negotiation successful
0620 11:42:57.222507 (+ 62us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":43}
I20260620 11:42:57.222704 14679 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 11:42:57.212710 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.234.254:44159 (local address 127.6.234.193:50381)
0620 11:42:57.212857 (+ 147us) negotiation.cc:107] Waiting for socket to connect
0620 11:42:57.212894 (+ 37us) client_negotiation.cc:174] Beginning negotiation
0620 11:42:57.213355 (+ 461us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 11:42:57.213617 (+ 262us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 11:42:57.213620 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 11:42:57.213978 (+ 358us) client_negotiation.cc:189] Negotiated authn=SASL
0620 11:42:57.214243 (+ 265us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 11:42:57.214249 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 11:42:57.215336 (+ 1087us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 11:42:57.215337 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 11:42:57.216683 (+ 1346us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 11:42:57.216686 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 11:42:57.217256 (+ 570us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 11:42:57.217257 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 11:42:57.217400 (+ 143us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 11:42:57.217924 (+ 524us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 11:42:57.217935 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0620 11:42:57.219815 (+ 1880us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 11:42:57.221487 (+ 1672us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 11:42:57.221491 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 11:42:57.221493 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0620 11:42:57.221685 (+ 192us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 11:42:57.221929 (+ 244us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 11:42:57.221930 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 11:42:57.221930 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0620 11:42:57.222021 (+ 91us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 11:42:57.222230 (+ 209us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 11:42:57.222234 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 11:42:57.222334 (+ 100us) client_negotiation.cc:769] Sending connection context
0620 11:42:57.222405 (+ 71us) client_negotiation.cc:240] Negotiation successful
0620 11:42:57.222512 (+ 107us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":114,"thread_start_us":83,"threads_started":1}
I20260620 11:42:57.223192 14786 heartbeater.cc:347] Connected to a master server at 127.6.234.254:44159
I20260620 11:42:57.223387 14786 heartbeater.cc:464] Registering TS with master...
I20260620 11:42:57.223568 14786 heartbeater.cc:511] Master 127.6.234.254:44159 requested a full tablet report, sending...
I20260620 11:42:57.224107 14583 ts_manager.cc:194] Registered new tserver with Master: 580c848b417d4ffbab6d865896c13de7 (127.6.234.193:44461)
I20260620 11:42:57.225016 14583 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.6.234.193@KRBTEST.COM'} at 127.6.234.193:50381
I20260620 11:42:57.225172 7083 external_mini_cluster.cc:1388] Started /tmp/dist-test-task1JRg7N/build/release/bin/kudu as pid 14655
I20260620 11:42:57.225276 7083 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task1JRg7N/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781955746769051-7083-0/minicluster-data/ts-0/wal/instance
I20260620 11:42:57.225508 7083 external_mini_cluster.cc:1428] Setting key d54ae585379b22b7cbcdeb155d3c0d41
I20260620 11:42:57.230144 14633 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 11:42:57.224222 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.6.234.193:60443 (local address 127.6.234.254:44159)
0620 11:42:57.224258 (+ 36us) server_negotiation.cc:206] Beginning negotiation
0620 11:42:57.224260 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 11:42:57.224395 (+ 135us) server_negotiation.cc:407] Connection header received
0620 11:42:57.224453 (+ 58us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 11:42:57.224454 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 11:42:57.224463 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 11:42:57.224507 (+ 44us) server_negotiation.cc:226] Negotiated authn=SASL
0620 11:42:57.224807 (+ 300us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 11:42:57.225419 (+ 612us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 11:42:57.226402 (+ 983us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 11:42:57.226577 (+ 175us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 11:42:57.226630 (+ 53us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 11:42:57.227571 (+ 941us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 11:42:57.227573 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 11:42:57.227573 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 11:42:57.227580 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0620 11:42:57.229027 (+ 1447us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 11:42:57.229410 (+ 383us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 11:42:57.229411 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 11:42:57.229411 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 11:42:57.229457 (+ 46us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 11:42:57.229643 (+ 186us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 11:42:57.229643 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 11:42:57.229644 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 11:42:57.229799 (+ 155us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 11:42:57.229853 (+ 54us) server_negotiation.cc:1035] Waiting for connection context
0620 11:42:57.230028 (+ 175us) server_negotiation.cc:299] Negotiation successful
0620 11:42:57.230086 (+ 58us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":19}
I20260620 11:42:57.230232 14792 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 11:42:57.224157 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.234.254:44159 (local address 127.6.234.193:60443)
0620 11:42:57.224316 (+ 159us) negotiation.cc:107] Waiting for socket to connect
0620 11:42:57.224326 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0620 11:42:57.224396 (+ 70us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 11:42:57.224511 (+ 115us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 11:42:57.224512 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 11:42:57.224692 (+ 180us) client_negotiation.cc:189] Negotiated authn=SASL
0620 11:42:57.224743 (+ 51us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 11:42:57.224745 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 11:42:57.225511 (+ 766us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 11:42:57.225512 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 11:42:57.226318 (+ 806us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 11:42:57.226321 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 11:42:57.226649 (+ 328us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 11:42:57.226650 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 11:42:57.226722 (+ 72us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 11:42:57.227094 (+ 372us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 11:42:57.227099 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0620 11:42:57.227484 (+ 385us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 11:42:57.229108 (+ 1624us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 11:42:57.229109 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 11:42:57.229109 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0620 11:42:57.229334 (+ 225us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 11:42:57.229517 (+ 183us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 11:42:57.229517 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 11:42:57.229518 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0620 11:42:57.229586 (+ 68us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 11:42:57.229874 (+ 288us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 11:42:57.229875 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 11:42:57.229926 (+ 51us) client_negotiation.cc:769] Sending connection context
0620 11:42:57.230003 (+ 77us) client_negotiation.cc:240] Negotiation successful
0620 11:42:57.230118 (+ 115us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":136,"mutex_wait_us":40,"thread_start_us":63,"threads_started":1}
WARNING: no policy specified for oryx/127.6.234.194@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.6.234.194@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.6.234.194 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task1JRg7N/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781955746769051-7083-0/krb5kdc/oryx_127.6.234.194.keytab.
Entry for principal oryx/127.6.234.194 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task1JRg7N/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781955746769051-7083-0/krb5kdc/oryx_127.6.234.194.keytab.
I20260620 11:42:57.249482 7083 external_mini_cluster.cc:1326] Running /tmp/dist-test-task1JRg7N/build/release/bin/kudu
/tmp/dist-test-task1JRg7N/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task1JRg7N/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781955746769051-7083-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-task1JRg7N/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781955746769051-7083-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task1JRg7N/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781955746769051-7083-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-task1JRg7N/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781955746769051-7083-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.6.234.194:0
--local_ip_for_outbound_sockets=127.6.234.194
--webserver_interface=127.6.234.194
--webserver_port=0
--tserver_master_addrs=127.6.234.254:44159
--keytab_file=/tmp/dist-test-task1JRg7N/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781955746769051-7083-0/krb5kdc/oryx_127.6.234.194.keytab
--principal=oryx/127.6.234.194
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.6.234.212:41127
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true with env {KRB5CCNAME=/tmp/dist-test-task1JRg7N/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781955746769051-7083-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task1JRg7N/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781955746769051-7083-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task1JRg7N/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781955746769051-7083-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 11:42:57.329895 14795 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 11:42:57.330084 14795 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 11:42:57.330108 14795 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 11:42:57.331179 14795 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 11:42:57.331210 14795 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 11:42:57.331229 14795 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260620 11:42:57.331245 14795 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.6.234.194
I20260620 11:42:57.332655 14795 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.6.234.212:41127
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task1JRg7N/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781955746769051-7083-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-task1JRg7N/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781955746769051-7083-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task1JRg7N/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781955746769051-7083-0/krb5kdc/oryx_127.6.234.194.keytab
--principal=oryx/127.6.234.194
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.6.234.194:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task1JRg7N/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781955746769051-7083-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.6.234.194
--webserver_port=0
--enable_txn_system_client_init=true
--tserver_master_addrs=127.6.234.254:44159
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.14795
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.6.234.194
--log_dir=/tmp/dist-test-task1JRg7N/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781955746769051-7083-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 20 Jun 2026 10:32:37 UTC on ccycloud.quasar-bgyofv.root.comops.site
build id 4457173
I20260620 11:42:57.332841 14795 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 11:42:57.333003 14795 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 11:42:57.337882 14800 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 11:42:57.337869 14803 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 11:42:57.337860 14801 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 11:42:57.338143 14795 server_base.cc:1034] running on GCE node
I20260620 11:42:57.338398 14795 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 11:42:57.338630 14795 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 11:42:57.339780 14795 hybrid_clock.cc:648] HybridClock initialized: now 1781955777339763 us; error 35 us; skew 500 ppm
Jun 20 11:42:57 dist-test-slave-cjc6 krb5kdc[14538](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781955777, etypes {rep=17 tkt=17 ses=17}, oryx/127.6.234.194@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 11:42:57.342274 14795 init.cc:377] Logged in from keytab as oryx/127.6.234.194@KRBTEST.COM (short username oryx)
I20260620 11:42:57.342680 14795 webserver.cc:460] Webserver started at http://127.6.234.194:35255/ using document root <none> and password file <none>
I20260620 11:42:57.342911 14795 fs_manager.cc:359] Metadata directory not provided
I20260620 11:42:57.342978 14795 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 11:42:57.343127 14795 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 11:42:57.344135 14795 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task1JRg7N/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781955746769051-7083-0/minicluster-data/ts-1/data/instance:
uuid: "82db96227ab34c6e93967a5951456933"
format_stamp: "Formatted at 2026-06-20 11:42:57 on dist-test-slave-cjc6"
server_key: "b8c07e02df4b6932777322aeb1bc90af"
server_key_iv: "e8aeefe38850948f7a0db8cbb5374dfd"
server_key_version: "encryptionkey@0"
I20260620 11:42:57.344498 14795 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task1JRg7N/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781955746769051-7083-0/minicluster-data/ts-1/wal/instance:
uuid: "82db96227ab34c6e93967a5951456933"
format_stamp: "Formatted at 2026-06-20 11:42:57 on dist-test-slave-cjc6"
server_key: "b8c07e02df4b6932777322aeb1bc90af"
server_key_iv: "e8aeefe38850948f7a0db8cbb5374dfd"
server_key_version: "encryptionkey@0"
I20260620 11:42:57.345865 14795 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260620 11:42:57.346624 14810 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 11:42:57.346805 14795 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260620 11:42:57.346900 14795 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task1JRg7N/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781955746769051-7083-0/minicluster-data/ts-1/data,/tmp/dist-test-task1JRg7N/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781955746769051-7083-0/minicluster-data/ts-1/wal
uuid: "82db96227ab34c6e93967a5951456933"
format_stamp: "Formatted at 2026-06-20 11:42:57 on dist-test-slave-cjc6"
server_key: "b8c07e02df4b6932777322aeb1bc90af"
server_key_iv: "e8aeefe38850948f7a0db8cbb5374dfd"
server_key_version: "encryptionkey@0"
I20260620 11:42:57.346985 14795 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task1JRg7N/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781955746769051-7083-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-task1JRg7N/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781955746769051-7083-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-task1JRg7N/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781955746769051-7083-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 11:42:57.361248 14795 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 11:42:57.363595 14795 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 11:42:57.363761 14795 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 11:42:57.364262 14795 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 11:42:57.364323 14795 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 11:42:57.364359 14795 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 11:42:57.364408 14795 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 11:42:57.371176 14795 rpc_server.cc:307] RPC server started. Bound to: 127.6.234.194:40195
I20260620 11:42:57.371244 14925 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.6.234.194:40195 every 8 connection(s)
I20260620 11:42:57.371524 14795 server_base.cc:1166] Dumped server information to /tmp/dist-test-task1JRg7N/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781955746769051-7083-0/minicluster-data/ts-1/data/info.pb
Jun 20 11:42:57 dist-test-slave-cjc6 krb5kdc[14538](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781955777, etypes {rep=17 tkt=17 ses=17}, oryx/127.6.234.194@KRBTEST.COM for oryx/127.6.234.254@KRBTEST.COM
I20260620 11:42:57.373932 7083 external_mini_cluster.cc:1388] Started /tmp/dist-test-task1JRg7N/build/release/bin/kudu as pid 14795
I20260620 11:42:57.374064 7083 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task1JRg7N/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781955746769051-7083-0/minicluster-data/ts-1/wal/instance
I20260620 11:42:57.374280 7083 external_mini_cluster.cc:1428] Setting key 92ea5428f56143185d5908849b96ba85
I20260620 11:42:57.374778 14633 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 11:42:57.364715 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.6.234.194:41601 (local address 127.6.234.254:44159)
0620 11:42:57.364790 (+ 75us) server_negotiation.cc:206] Beginning negotiation
0620 11:42:57.364792 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 11:42:57.365022 (+ 230us) server_negotiation.cc:407] Connection header received
0620 11:42:57.365569 (+ 547us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 11:42:57.365571 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 11:42:57.365586 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 11:42:57.365677 (+ 91us) server_negotiation.cc:226] Negotiated authn=SASL
0620 11:42:57.366547 (+ 870us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 11:42:57.367376 (+ 829us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 11:42:57.369061 (+ 1685us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 11:42:57.369369 (+ 308us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 11:42:57.369428 (+ 59us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 11:42:57.372107 (+ 2679us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 11:42:57.372109 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 11:42:57.372109 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 11:42:57.372117 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0620 11:42:57.373270 (+ 1153us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 11:42:57.373723 (+ 453us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 11:42:57.373724 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 11:42:57.373725 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 11:42:57.373784 (+ 59us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 11:42:57.374046 (+ 262us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 11:42:57.374047 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 11:42:57.374047 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 11:42:57.374198 (+ 151us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 11:42:57.374282 (+ 84us) server_negotiation.cc:1035] Waiting for connection context
0620 11:42:57.374602 (+ 320us) server_negotiation.cc:299] Negotiation successful
0620 11:42:57.374689 (+ 87us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":47}
I20260620 11:42:57.374950 14819 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 11:42:57.364683 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.234.254:44159 (local address 127.6.234.194:41601)
0620 11:42:57.364878 (+ 195us) negotiation.cc:107] Waiting for socket to connect
0620 11:42:57.364918 (+ 40us) client_negotiation.cc:174] Beginning negotiation
0620 11:42:57.365428 (+ 510us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 11:42:57.365783 (+ 355us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 11:42:57.365787 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 11:42:57.366176 (+ 389us) client_negotiation.cc:189] Negotiated authn=SASL
0620 11:42:57.366457 (+ 281us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 11:42:57.366463 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 11:42:57.367484 (+ 1021us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 11:42:57.367486 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 11:42:57.368958 (+ 1472us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 11:42:57.368962 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 11:42:57.369461 (+ 499us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 11:42:57.369462 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 11:42:57.369625 (+ 163us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 11:42:57.370207 (+ 582us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 11:42:57.370218 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0620 11:42:57.372009 (+ 1791us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 11:42:57.373376 (+ 1367us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 11:42:57.373382 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 11:42:57.373384 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0620 11:42:57.373605 (+ 221us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 11:42:57.373865 (+ 260us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 11:42:57.373866 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 11:42:57.373866 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0620 11:42:57.373977 (+ 111us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 11:42:57.374282 (+ 305us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 11:42:57.374290 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 11:42:57.374453 (+ 163us) client_negotiation.cc:769] Sending connection context
0620 11:42:57.374561 (+ 108us) client_negotiation.cc:240] Negotiation successful
0620 11:42:57.374714 (+ 153us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":157,"thread_start_us":84,"threads_started":1}
I20260620 11:42:57.376643 14926 heartbeater.cc:347] Connected to a master server at 127.6.234.254:44159
I20260620 11:42:57.376744 14926 heartbeater.cc:464] Registering TS with master...
I20260620 11:42:57.376924 14926 heartbeater.cc:511] Master 127.6.234.254:44159 requested a full tablet report, sending...
I20260620 11:42:57.377288 14582 ts_manager.cc:194] Registered new tserver with Master: 82db96227ab34c6e93967a5951456933 (127.6.234.194:40195)
I20260620 11:42:57.377640 14582 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.6.234.194@KRBTEST.COM'} at 127.6.234.194:41601
I20260620 11:42:57.383039 14633 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 11:42:57.376452 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.6.234.194:42607 (local address 127.6.234.254:44159)
0620 11:42:57.376494 (+ 42us) server_negotiation.cc:206] Beginning negotiation
0620 11:42:57.376496 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 11:42:57.376792 (+ 296us) server_negotiation.cc:407] Connection header received
0620 11:42:57.376894 (+ 102us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 11:42:57.376896 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 11:42:57.376908 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 11:42:57.376970 (+ 62us) server_negotiation.cc:226] Negotiated authn=SASL
0620 11:42:57.377318 (+ 348us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 11:42:57.378089 (+ 771us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 11:42:57.379228 (+ 1139us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 11:42:57.379435 (+ 207us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 11:42:57.379487 (+ 52us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 11:42:57.380523 (+ 1036us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 11:42:57.380525 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 11:42:57.380525 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 11:42:57.380530 (+ 5us) server_negotiation.cc:922] calling sasl_server_start()
0620 11:42:57.381938 (+ 1408us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 11:42:57.382308 (+ 370us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 11:42:57.382309 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 11:42:57.382309 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 11:42:57.382364 (+ 55us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 11:42:57.382567 (+ 203us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 11:42:57.382568 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 11:42:57.382568 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 11:42:57.382695 (+ 127us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 11:42:57.382745 (+ 50us) server_negotiation.cc:1035] Waiting for connection context
0620 11:42:57.382928 (+ 183us) server_negotiation.cc:299] Negotiation successful
0620 11:42:57.382981 (+ 53us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":20}
I20260620 11:42:57.383141 14933 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 11:42:57.376533 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.234.254:44159 (local address 127.6.234.194:42607)
0620 11:42:57.376727 (+ 194us) negotiation.cc:107] Waiting for socket to connect
0620 11:42:57.376740 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0620 11:42:57.376804 (+ 64us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 11:42:57.376975 (+ 171us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 11:42:57.376976 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 11:42:57.377175 (+ 199us) client_negotiation.cc:189] Negotiated authn=SASL
0620 11:42:57.377239 (+ 64us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 11:42:57.377241 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 11:42:57.378185 (+ 944us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 11:42:57.378186 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 11:42:57.379133 (+ 947us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 11:42:57.379136 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 11:42:57.379514 (+ 378us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 11:42:57.379515 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 11:42:57.379583 (+ 68us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 11:42:57.379990 (+ 407us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 11:42:57.379996 (+ 6us) client_negotiation.cc:656] calling sasl_client_start()
0620 11:42:57.380439 (+ 443us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 11:42:57.382016 (+ 1577us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 11:42:57.382017 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 11:42:57.382017 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0620 11:42:57.382228 (+ 211us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 11:42:57.382429 (+ 201us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 11:42:57.382430 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 11:42:57.382430 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0620 11:42:57.382502 (+ 72us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 11:42:57.382765 (+ 263us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 11:42:57.382766 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 11:42:57.382820 (+ 54us) client_negotiation.cc:769] Sending connection context
0620 11:42:57.382913 (+ 93us) client_negotiation.cc:240] Negotiation successful
0620 11:42:57.383027 (+ 114us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":119,"thread_start_us":73,"threads_started":1}
WARNING: no policy specified for oryx/127.6.234.195@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.6.234.195@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.6.234.195 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task1JRg7N/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781955746769051-7083-0/krb5kdc/oryx_127.6.234.195.keytab.
Entry for principal oryx/127.6.234.195 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task1JRg7N/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781955746769051-7083-0/krb5kdc/oryx_127.6.234.195.keytab.
I20260620 11:42:57.400159 7083 external_mini_cluster.cc:1326] Running /tmp/dist-test-task1JRg7N/build/release/bin/kudu
/tmp/dist-test-task1JRg7N/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task1JRg7N/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781955746769051-7083-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-task1JRg7N/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781955746769051-7083-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task1JRg7N/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781955746769051-7083-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-task1JRg7N/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781955746769051-7083-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.6.234.195:0
--local_ip_for_outbound_sockets=127.6.234.195
--webserver_interface=127.6.234.195
--webserver_port=0
--tserver_master_addrs=127.6.234.254:44159
--keytab_file=/tmp/dist-test-task1JRg7N/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781955746769051-7083-0/krb5kdc/oryx_127.6.234.195.keytab
--principal=oryx/127.6.234.195
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.6.234.212:41127
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true with env {KRB5CCNAME=/tmp/dist-test-task1JRg7N/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781955746769051-7083-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task1JRg7N/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781955746769051-7083-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task1JRg7N/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781955746769051-7083-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 11:42:57.480535 14935 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 11:42:57.480722 14935 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 11:42:57.480746 14935 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 11:42:57.481915 14935 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 11:42:57.481957 14935 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 11:42:57.481978 14935 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260620 11:42:57.482033 14935 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.6.234.195
I20260620 11:42:57.483521 14935 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.6.234.212:41127
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task1JRg7N/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781955746769051-7083-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-task1JRg7N/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781955746769051-7083-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task1JRg7N/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781955746769051-7083-0/krb5kdc/oryx_127.6.234.195.keytab
--principal=oryx/127.6.234.195
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.6.234.195:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task1JRg7N/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781955746769051-7083-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.6.234.195
--webserver_port=0
--enable_txn_system_client_init=true
--tserver_master_addrs=127.6.234.254:44159
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.14935
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.6.234.195
--log_dir=/tmp/dist-test-task1JRg7N/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781955746769051-7083-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 20 Jun 2026 10:32:37 UTC on ccycloud.quasar-bgyofv.root.comops.site
build id 4457173
I20260620 11:42:57.483763 14935 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 11:42:57.484016 14935 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 11:42:57.488929 14941 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 11:42:57.488914 14940 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 11:42:57.489007 14935 server_base.cc:1034] running on GCE node
W20260620 11:42:57.488936 14943 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 11:42:57.489297 14935 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 11:42:57.489493 14935 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 11:42:57.490636 14935 hybrid_clock.cc:648] HybridClock initialized: now 1781955777490627 us; error 37 us; skew 500 ppm
Jun 20 11:42:57 dist-test-slave-cjc6 krb5kdc[14538](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781955777, etypes {rep=17 tkt=17 ses=17}, oryx/127.6.234.195@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 11:42:57.492993 14935 init.cc:377] Logged in from keytab as oryx/127.6.234.195@KRBTEST.COM (short username oryx)
I20260620 11:42:57.493408 14935 webserver.cc:460] Webserver started at http://127.6.234.195:34795/ using document root <none> and password file <none>
I20260620 11:42:57.493649 14935 fs_manager.cc:359] Metadata directory not provided
I20260620 11:42:57.493757 14935 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 11:42:57.493907 14935 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 11:42:57.494964 14935 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task1JRg7N/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781955746769051-7083-0/minicluster-data/ts-2/data/instance:
uuid: "d4e3cd899940498d83eba8fc5ed6b9a2"
format_stamp: "Formatted at 2026-06-20 11:42:57 on dist-test-slave-cjc6"
server_key: "a846da6651a4f4b289445733fa399968"
server_key_iv: "41910e13deca80ddad73f25af852f1e1"
server_key_version: "encryptionkey@0"
I20260620 11:42:57.495309 14935 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task1JRg7N/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781955746769051-7083-0/minicluster-data/ts-2/wal/instance:
uuid: "d4e3cd899940498d83eba8fc5ed6b9a2"
format_stamp: "Formatted at 2026-06-20 11:42:57 on dist-test-slave-cjc6"
server_key: "a846da6651a4f4b289445733fa399968"
server_key_iv: "41910e13deca80ddad73f25af852f1e1"
server_key_version: "encryptionkey@0"
I20260620 11:42:57.496615 14935 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.003s
I20260620 11:42:57.497330 14950 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 11:42:57.497509 14935 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.001s
I20260620 11:42:57.497607 14935 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task1JRg7N/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781955746769051-7083-0/minicluster-data/ts-2/data,/tmp/dist-test-task1JRg7N/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781955746769051-7083-0/minicluster-data/ts-2/wal
uuid: "d4e3cd899940498d83eba8fc5ed6b9a2"
format_stamp: "Formatted at 2026-06-20 11:42:57 on dist-test-slave-cjc6"
server_key: "a846da6651a4f4b289445733fa399968"
server_key_iv: "41910e13deca80ddad73f25af852f1e1"
server_key_version: "encryptionkey@0"
I20260620 11:42:57.497731 14935 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task1JRg7N/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781955746769051-7083-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-task1JRg7N/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781955746769051-7083-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-task1JRg7N/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781955746769051-7083-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 11:42:57.505905 14935 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 11:42:57.507721 14935 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 11:42:57.507871 14935 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 11:42:57.508333 14935 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 11:42:57.508397 14935 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 11:42:57.508461 14935 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 11:42:57.508489 14935 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 11:42:57.515295 14935 rpc_server.cc:307] RPC server started. Bound to: 127.6.234.195:38309
Jun 20 11:42:57 dist-test-slave-cjc6 krb5kdc[14538](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781955777, etypes {rep=17 tkt=17 ses=17}, oryx/127.6.234.195@KRBTEST.COM for oryx/127.6.234.254@KRBTEST.COM
I20260620 11:42:57.515363 15065 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.6.234.195:38309 every 8 connection(s)
I20260620 11:42:57.515676 14935 server_base.cc:1166] Dumped server information to /tmp/dist-test-task1JRg7N/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781955746769051-7083-0/minicluster-data/ts-2/data/info.pb
I20260620 11:42:57.518348 14633 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 11:42:57.508786 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.6.234.195:40673 (local address 127.6.234.254:44159)
0620 11:42:57.508853 (+ 67us) server_negotiation.cc:206] Beginning negotiation
0620 11:42:57.508856 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0620 11:42:57.509042 (+ 186us) server_negotiation.cc:407] Connection header received
0620 11:42:57.509488 (+ 446us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 11:42:57.509490 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 11:42:57.509505 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 11:42:57.509587 (+ 82us) server_negotiation.cc:226] Negotiated authn=SASL
0620 11:42:57.510374 (+ 787us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 11:42:57.511211 (+ 837us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 11:42:57.512819 (+ 1608us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 11:42:57.513126 (+ 307us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 11:42:57.513205 (+ 79us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 11:42:57.515906 (+ 2701us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 11:42:57.515907 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 11:42:57.515907 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 11:42:57.515914 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0620 11:42:57.517160 (+ 1246us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 11:42:57.517519 (+ 359us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 11:42:57.517520 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 11:42:57.517520 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 11:42:57.517569 (+ 49us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 11:42:57.517842 (+ 273us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 11:42:57.517843 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 11:42:57.517843 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 11:42:57.517942 (+ 99us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 11:42:57.517994 (+ 52us) server_negotiation.cc:1035] Waiting for connection context
0620 11:42:57.518223 (+ 229us) server_negotiation.cc:299] Negotiation successful
0620 11:42:57.518279 (+ 56us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":41}
I20260620 11:42:57.518479 14959 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 11:42:57.508780 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.234.254:44159 (local address 127.6.234.195:40673)
0620 11:42:57.508915 (+ 135us) negotiation.cc:107] Waiting for socket to connect
0620 11:42:57.508953 (+ 38us) client_negotiation.cc:174] Beginning negotiation
0620 11:42:57.509397 (+ 444us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 11:42:57.509632 (+ 235us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 11:42:57.509636 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 11:42:57.509999 (+ 363us) client_negotiation.cc:189] Negotiated authn=SASL
0620 11:42:57.510268 (+ 269us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 11:42:57.510274 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 11:42:57.511337 (+ 1063us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 11:42:57.511338 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 11:42:57.512696 (+ 1358us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 11:42:57.512698 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 11:42:57.513235 (+ 537us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 11:42:57.513237 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 11:42:57.513396 (+ 159us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 11:42:57.513952 (+ 556us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 11:42:57.513963 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0620 11:42:57.515815 (+ 1852us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 11:42:57.517239 (+ 1424us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 11:42:57.517244 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 11:42:57.517246 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0620 11:42:57.517453 (+ 207us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 11:42:57.517626 (+ 173us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 11:42:57.517626 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 11:42:57.517627 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0620 11:42:57.517783 (+ 156us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 11:42:57.517999 (+ 216us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 11:42:57.518003 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 11:42:57.518108 (+ 105us) client_negotiation.cc:769] Sending connection context
0620 11:42:57.518183 (+ 75us) client_negotiation.cc:240] Negotiation successful
0620 11:42:57.518301 (+ 118us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":98,"thread_start_us":73,"threads_started":1}
I20260620 11:42:57.518981 15066 heartbeater.cc:347] Connected to a master server at 127.6.234.254:44159
I20260620 11:42:57.519058 15066 heartbeater.cc:464] Registering TS with master...
I20260620 11:42:57.519223 15066 heartbeater.cc:511] Master 127.6.234.254:44159 requested a full tablet report, sending...
I20260620 11:42:57.519599 14583 ts_manager.cc:194] Registered new tserver with Master: d4e3cd899940498d83eba8fc5ed6b9a2 (127.6.234.195:38309)
I20260620 11:42:57.520125 14583 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.6.234.195@KRBTEST.COM'} at 127.6.234.195:40673
I20260620 11:42:57.524712 7083 external_mini_cluster.cc:1388] Started /tmp/dist-test-task1JRg7N/build/release/bin/kudu as pid 14935
I20260620 11:42:57.524837 7083 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task1JRg7N/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781955746769051-7083-0/minicluster-data/ts-2/wal/instance
I20260620 11:42:57.525032 7083 external_mini_cluster.cc:1428] Setting key 826cf04c7b8ede98a36e7d19d013b342
I20260620 11:42:57.525214 14633 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 11:42:57.520064 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.6.234.195:53875 (local address 127.6.234.254:44159)
0620 11:42:57.520100 (+ 36us) server_negotiation.cc:206] Beginning negotiation
0620 11:42:57.520102 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 11:42:57.520217 (+ 115us) server_negotiation.cc:407] Connection header received
0620 11:42:57.520238 (+ 21us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 11:42:57.520239 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 11:42:57.520247 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 11:42:57.520307 (+ 60us) server_negotiation.cc:226] Negotiated authn=SASL
0620 11:42:57.520780 (+ 473us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 11:42:57.521221 (+ 441us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 11:42:57.521958 (+ 737us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 11:42:57.522133 (+ 175us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 11:42:57.522187 (+ 54us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 11:42:57.522927 (+ 740us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 11:42:57.522929 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 11:42:57.522929 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 11:42:57.522934 (+ 5us) server_negotiation.cc:922] calling sasl_server_start()
0620 11:42:57.524151 (+ 1217us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 11:42:57.524443 (+ 292us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 11:42:57.524444 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 11:42:57.524444 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 11:42:57.524489 (+ 45us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 11:42:57.524687 (+ 198us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 11:42:57.524687 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 11:42:57.524687 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 11:42:57.524798 (+ 111us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 11:42:57.524856 (+ 58us) server_negotiation.cc:1035] Waiting for connection context
0620 11:42:57.525068 (+ 212us) server_negotiation.cc:299] Negotiation successful
0620 11:42:57.525133 (+ 65us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":17}
I20260620 11:42:57.525231 15072 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 11:42:57.519993 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.234.254:44159 (local address 127.6.234.195:53875)
0620 11:42:57.520116 (+ 123us) negotiation.cc:107] Waiting for socket to connect
0620 11:42:57.520127 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0620 11:42:57.520182 (+ 55us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 11:42:57.520445 (+ 263us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 11:42:57.520445 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 11:42:57.520635 (+ 190us) client_negotiation.cc:189] Negotiated authn=SASL
0620 11:42:57.520704 (+ 69us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 11:42:57.520706 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 11:42:57.521304 (+ 598us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 11:42:57.521305 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 11:42:57.521874 (+ 569us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 11:42:57.521876 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 11:42:57.522195 (+ 319us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 11:42:57.522196 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 11:42:57.522239 (+ 43us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 11:42:57.522586 (+ 347us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 11:42:57.522591 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0620 11:42:57.522856 (+ 265us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 11:42:57.524217 (+ 1361us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 11:42:57.524218 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 11:42:57.524218 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0620 11:42:57.524383 (+ 165us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 11:42:57.524551 (+ 168us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 11:42:57.524552 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 11:42:57.524552 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0620 11:42:57.524617 (+ 65us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 11:42:57.524900 (+ 283us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 11:42:57.524901 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 11:42:57.524950 (+ 49us) client_negotiation.cc:769] Sending connection context
0620 11:42:57.525027 (+ 77us) client_negotiation.cc:240] Negotiation successful
0620 11:42:57.525128 (+ 101us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":96,"thread_start_us":60,"threads_started":1}
I20260620 11:42:57.525995 7083 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
Jun 20 11:42:57 dist-test-slave-cjc6 krb5kdc[14538](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781955777, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-user@KRBTEST.COM:
Jun 20 11:42:57 dist-test-slave-cjc6 krb5kdc[14538](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, test-user@KRBTEST.COM for kudu/127.6.234.254@KRBTEST.COM, Server not found in Kerberos database
Jun 20 11:42:57 dist-test-slave-cjc6 krb5kdc[14538](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, test-user@KRBTEST.COM for kudu/127.6.234.254@KRBTEST.COM, Server not found in Kerberos database
W20260620 11:42:57.546187 14633 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 11:42:57.541175 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46532 (local address 127.6.234.254:44159)
0620 11:42:57.541238 (+ 63us) server_negotiation.cc:206] Beginning negotiation
0620 11:42:57.541239 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0620 11:42:57.541248 (+ 9us) server_negotiation.cc:407] Connection header received
0620 11:42:57.541271 (+ 23us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 11:42:57.541272 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 11:42:57.541280 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 11:42:57.541335 (+ 55us) server_negotiation.cc:226] Negotiated authn=SASL
0620 11:42:57.541939 (+ 604us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 11:42:57.542425 (+ 486us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 11:42:57.543107 (+ 682us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 11:42:57.543280 (+ 173us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 11:42:57.543330 (+ 50us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 11:42:57.546092 (+ 2762us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:46532: BlockingRecv error: recv got EOF from 127.0.0.1:46532 (error 108)
Metrics: {"server-negotiator.queue_time_us":34}
Jun 20 11:42:57 dist-test-slave-cjc6 krb5kdc[14538](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781955777, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for oryx/127.6.234.254@KRBTEST.COM
I20260620 11:42:57.553138 14633 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 11:42:57.546966 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46544 (local address 127.6.234.254:44159)
0620 11:42:57.547006 (+ 40us) server_negotiation.cc:206] Beginning negotiation
0620 11:42:57.547007 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0620 11:42:57.547095 (+ 88us) server_negotiation.cc:407] Connection header received
0620 11:42:57.547141 (+ 46us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 11:42:57.547141 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 11:42:57.547149 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 11:42:57.547197 (+ 48us) server_negotiation.cc:226] Negotiated authn=SASL
0620 11:42:57.547579 (+ 382us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 11:42:57.548013 (+ 434us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 11:42:57.548617 (+ 604us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 11:42:57.548783 (+ 166us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 11:42:57.548833 (+ 50us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 11:42:57.550826 (+ 1993us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 11:42:57.550828 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 11:42:57.550828 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 11:42:57.550834 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0620 11:42:57.552005 (+ 1171us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 11:42:57.552308 (+ 303us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 11:42:57.552309 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 11:42:57.552310 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 11:42:57.552372 (+ 62us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 11:42:57.552597 (+ 225us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 11:42:57.552598 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 11:42:57.552598 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 11:42:57.552711 (+ 113us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 11:42:57.552810 (+ 99us) server_negotiation.cc:1035] Waiting for connection context
0620 11:42:57.552975 (+ 165us) server_negotiation.cc:299] Negotiation successful
0620 11:42:57.553043 (+ 68us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":20}
I20260620 11:42:57.554786 14583 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:46544:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260620 11:42:57.556270 14583 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260620 11:42:57.562624 15088 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 11:42:57.558721 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39656 (local address 127.6.234.194:40195)
0620 11:42:57.558892 (+ 171us) server_negotiation.cc:206] Beginning negotiation
0620 11:42:57.558897 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 11:42:57.559017 (+ 120us) server_negotiation.cc:407] Connection header received
0620 11:42:57.559097 (+ 80us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 11:42:57.559100 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 11:42:57.559117 (+ 17us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 11:42:57.559197 (+ 80us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 11:42:57.559411 (+ 214us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 11:42:57.560524 (+ 1113us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 11:42:57.561746 (+ 1222us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 11:42:57.562230 (+ 484us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 11:42:57.562327 (+ 97us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 11:42:57.562407 (+ 80us) server_negotiation.cc:1035] Waiting for connection context
0620 11:42:57.562495 (+ 88us) server_negotiation.cc:299] Negotiation successful
0620 11:42:57.562531 (+ 36us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":136,"thread_start_us":76,"threads_started":1}
I20260620 11:42:57.562611 15085 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 11:42:57.558441 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.234.193:44461 (local address 127.0.0.1:40862)
0620 11:42:57.558550 (+ 109us) negotiation.cc:107] Waiting for socket to connect
0620 11:42:57.558587 (+ 37us) client_negotiation.cc:174] Beginning negotiation
0620 11:42:57.558692 (+ 105us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 11:42:57.559144 (+ 452us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 11:42:57.559147 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 11:42:57.559150 (+ 3us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 11:42:57.559217 (+ 67us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 11:42:57.559219 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 11:42:57.560210 (+ 991us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 11:42:57.560211 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 11:42:57.561362 (+ 1151us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 11:42:57.561364 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 11:42:57.562304 (+ 940us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 11:42:57.562305 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 11:42:57.562386 (+ 81us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 11:42:57.562402 (+ 16us) client_negotiation.cc:769] Sending connection context
0620 11:42:57.562453 (+ 51us) client_negotiation.cc:240] Negotiation successful
0620 11:42:57.562502 (+ 49us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":86,"thread_start_us":67,"threads_started":1}
I20260620 11:42:57.562949 15086 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 11:42:57.558476 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.234.195:38309 (local address 127.0.0.1:44454)
0620 11:42:57.559105 (+ 629us) negotiation.cc:107] Waiting for socket to connect
0620 11:42:57.559114 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0620 11:42:57.559152 (+ 38us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 11:42:57.559472 (+ 320us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 11:42:57.559473 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 11:42:57.559475 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 11:42:57.559508 (+ 33us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 11:42:57.559509 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 11:42:57.560676 (+ 1167us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 11:42:57.560677 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 11:42:57.561722 (+ 1045us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 11:42:57.561725 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 11:42:57.562687 (+ 962us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 11:42:57.562688 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 11:42:57.562792 (+ 104us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 11:42:57.562796 (+ 4us) client_negotiation.cc:769] Sending connection context
0620 11:42:57.562848 (+ 52us) client_negotiation.cc:240] Negotiation successful
0620 11:42:57.562889 (+ 41us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":619,"mutex_wait_us":13,"thread_start_us":41,"threads_started":1}
I20260620 11:42:57.563149 15089 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 11:42:57.558734 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40862 (local address 127.6.234.193:44461)
0620 11:42:57.558951 (+ 217us) server_negotiation.cc:206] Beginning negotiation
0620 11:42:57.558956 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 11:42:57.558967 (+ 11us) server_negotiation.cc:407] Connection header received
0620 11:42:57.559008 (+ 41us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 11:42:57.559010 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 11:42:57.559024 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 11:42:57.559295 (+ 271us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 11:42:57.559368 (+ 73us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 11:42:57.560126 (+ 758us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 11:42:57.561453 (+ 1327us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 11:42:57.562206 (+ 753us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 11:42:57.562884 (+ 678us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 11:42:57.562967 (+ 83us) server_negotiation.cc:1035] Waiting for connection context
0620 11:42:57.562993 (+ 26us) server_negotiation.cc:299] Negotiation successful
0620 11:42:57.563041 (+ 48us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":178,"thread_start_us":116,"threads_started":1}
I20260620 11:42:57.563194 15090 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 11:42:57.558913 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44454 (local address 127.6.234.195:38309)
0620 11:42:57.559103 (+ 190us) server_negotiation.cc:206] Beginning negotiation
0620 11:42:57.559274 (+ 171us) server_negotiation.cc:399] Waiting for connection header
0620 11:42:57.559286 (+ 12us) server_negotiation.cc:407] Connection header received
0620 11:42:57.559323 (+ 37us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 11:42:57.559325 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 11:42:57.559341 (+ 16us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 11:42:57.559413 (+ 72us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 11:42:57.559586 (+ 173us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 11:42:57.560616 (+ 1030us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 11:42:57.561811 (+ 1195us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 11:42:57.562520 (+ 709us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 11:42:57.562585 (+ 65us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 11:42:57.562654 (+ 69us) server_negotiation.cc:1035] Waiting for connection context
0620 11:42:57.563054 (+ 400us) server_negotiation.cc:299] Negotiation successful
0620 11:42:57.563092 (+ 38us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":161,"thread_start_us":89,"threads_started":1}
I20260620 11:42:57.562618 15087 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 11:42:57.558476 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.234.194:40195 (local address 127.0.0.1:39656)
0620 11:42:57.558937 (+ 461us) negotiation.cc:107] Waiting for socket to connect
0620 11:42:57.558947 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0620 11:42:57.559027 (+ 80us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 11:42:57.559262 (+ 235us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 11:42:57.559262 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 11:42:57.559265 (+ 3us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 11:42:57.559326 (+ 61us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 11:42:57.559328 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 11:42:57.560612 (+ 1284us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 11:42:57.560613 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 11:42:57.561630 (+ 1017us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 11:42:57.561632 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 11:42:57.562331 (+ 699us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 11:42:57.562332 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 11:42:57.562400 (+ 68us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 11:42:57.562405 (+ 5us) client_negotiation.cc:769] Sending connection context
0620 11:42:57.562477 (+ 72us) client_negotiation.cc:240] Negotiation successful
0620 11:42:57.562520 (+ 43us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":450,"mutex_wait_us":32,"thread_start_us":56,"threads_started":1}
I20260620 11:42:57.563843 14720 tablet_service.cc:1467] Processing CreateTablet for tablet ffcee0b5b0e648df9c9ad247ad77c93e (DEFAULT_TABLE table=test-table [id=9c7e9a98c7f8401c8edc7e99004e722b]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 11:42:57.563843 15000 tablet_service.cc:1467] Processing CreateTablet for tablet ffcee0b5b0e648df9c9ad247ad77c93e (DEFAULT_TABLE table=test-table [id=9c7e9a98c7f8401c8edc7e99004e722b]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 11:42:57.563969 14860 tablet_service.cc:1467] Processing CreateTablet for tablet ffcee0b5b0e648df9c9ad247ad77c93e (DEFAULT_TABLE table=test-table [id=9c7e9a98c7f8401c8edc7e99004e722b]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 11:42:57.564196 14720 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ffcee0b5b0e648df9c9ad247ad77c93e. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 11:42:57.564196 15000 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ffcee0b5b0e648df9c9ad247ad77c93e. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 11:42:57.564216 14860 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ffcee0b5b0e648df9c9ad247ad77c93e. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 11:42:57.567426 15091 tablet_bootstrap.cc:492] T ffcee0b5b0e648df9c9ad247ad77c93e P 82db96227ab34c6e93967a5951456933: Bootstrap starting.
I20260620 11:42:57.567785 15092 tablet_bootstrap.cc:492] T ffcee0b5b0e648df9c9ad247ad77c93e P d4e3cd899940498d83eba8fc5ed6b9a2: Bootstrap starting.
I20260620 11:42:57.568020 15093 tablet_bootstrap.cc:492] T ffcee0b5b0e648df9c9ad247ad77c93e P 580c848b417d4ffbab6d865896c13de7: Bootstrap starting.
I20260620 11:42:57.568195 15091 tablet_bootstrap.cc:654] T ffcee0b5b0e648df9c9ad247ad77c93e P 82db96227ab34c6e93967a5951456933: Neither blocks nor log segments found. Creating new log.
I20260620 11:42:57.568392 15092 tablet_bootstrap.cc:654] T ffcee0b5b0e648df9c9ad247ad77c93e P d4e3cd899940498d83eba8fc5ed6b9a2: Neither blocks nor log segments found. Creating new log.
I20260620 11:42:57.568537 15091 log.cc:824] T ffcee0b5b0e648df9c9ad247ad77c93e P 82db96227ab34c6e93967a5951456933: Log is configured to *not* fsync() on all Append() calls
I20260620 11:42:57.568657 15093 tablet_bootstrap.cc:654] T ffcee0b5b0e648df9c9ad247ad77c93e P 580c848b417d4ffbab6d865896c13de7: Neither blocks nor log segments found. Creating new log.
I20260620 11:42:57.568711 15092 log.cc:824] T ffcee0b5b0e648df9c9ad247ad77c93e P d4e3cd899940498d83eba8fc5ed6b9a2: Log is configured to *not* fsync() on all Append() calls
I20260620 11:42:57.568974 15093 log.cc:824] T ffcee0b5b0e648df9c9ad247ad77c93e P 580c848b417d4ffbab6d865896c13de7: Log is configured to *not* fsync() on all Append() calls
I20260620 11:42:57.569376 15091 tablet_bootstrap.cc:492] T ffcee0b5b0e648df9c9ad247ad77c93e P 82db96227ab34c6e93967a5951456933: No bootstrap required, opened a new log
I20260620 11:42:57.569432 15092 tablet_bootstrap.cc:492] T ffcee0b5b0e648df9c9ad247ad77c93e P d4e3cd899940498d83eba8fc5ed6b9a2: No bootstrap required, opened a new log
I20260620 11:42:57.569463 15091 ts_tablet_manager.cc:1397] T ffcee0b5b0e648df9c9ad247ad77c93e P 82db96227ab34c6e93967a5951456933: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260620 11:42:57.569516 15092 ts_tablet_manager.cc:1397] T ffcee0b5b0e648df9c9ad247ad77c93e P d4e3cd899940498d83eba8fc5ed6b9a2: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260620 11:42:57.570268 15093 tablet_bootstrap.cc:492] T ffcee0b5b0e648df9c9ad247ad77c93e P 580c848b417d4ffbab6d865896c13de7: No bootstrap required, opened a new log
I20260620 11:42:57.570338 15093 ts_tablet_manager.cc:1397] T ffcee0b5b0e648df9c9ad247ad77c93e P 580c848b417d4ffbab6d865896c13de7: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260620 11:42:57.571118 15092 raft_consensus.cc:348] T ffcee0b5b0e648df9c9ad247ad77c93e P d4e3cd899940498d83eba8fc5ed6b9a2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "580c848b417d4ffbab6d865896c13de7" member_type: VOTER last_known_addr { host: "127.6.234.193" port: 44461 } } peers { permanent_uuid: "82db96227ab34c6e93967a5951456933" member_type: VOTER last_known_addr { host: "127.6.234.194" port: 40195 } } peers { permanent_uuid: "d4e3cd899940498d83eba8fc5ed6b9a2" member_type: VOTER last_known_addr { host: "127.6.234.195" port: 38309 } }
I20260620 11:42:57.571269 15092 raft_consensus.cc:374] T ffcee0b5b0e648df9c9ad247ad77c93e P d4e3cd899940498d83eba8fc5ed6b9a2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 11:42:57.571311 15092 raft_consensus.cc:729] T ffcee0b5b0e648df9c9ad247ad77c93e P d4e3cd899940498d83eba8fc5ed6b9a2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d4e3cd899940498d83eba8fc5ed6b9a2, State: Initialized, Role: FOLLOWER
I20260620 11:42:57.571435 15092 consensus_queue.cc:260] T ffcee0b5b0e648df9c9ad247ad77c93e P d4e3cd899940498d83eba8fc5ed6b9a2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "580c848b417d4ffbab6d865896c13de7" member_type: VOTER last_known_addr { host: "127.6.234.193" port: 44461 } } peers { permanent_uuid: "82db96227ab34c6e93967a5951456933" member_type: VOTER last_known_addr { host: "127.6.234.194" port: 40195 } } peers { permanent_uuid: "d4e3cd899940498d83eba8fc5ed6b9a2" member_type: VOTER last_known_addr { host: "127.6.234.195" port: 38309 } }
I20260620 11:42:57.571465 15091 raft_consensus.cc:348] T ffcee0b5b0e648df9c9ad247ad77c93e P 82db96227ab34c6e93967a5951456933 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "580c848b417d4ffbab6d865896c13de7" member_type: VOTER last_known_addr { host: "127.6.234.193" port: 44461 } } peers { permanent_uuid: "82db96227ab34c6e93967a5951456933" member_type: VOTER last_known_addr { host: "127.6.234.194" port: 40195 } } peers { permanent_uuid: "d4e3cd899940498d83eba8fc5ed6b9a2" member_type: VOTER last_known_addr { host: "127.6.234.195" port: 38309 } }
I20260620 11:42:57.571627 15091 raft_consensus.cc:374] T ffcee0b5b0e648df9c9ad247ad77c93e P 82db96227ab34c6e93967a5951456933 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 11:42:57.571674 15091 raft_consensus.cc:729] T ffcee0b5b0e648df9c9ad247ad77c93e P 82db96227ab34c6e93967a5951456933 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 82db96227ab34c6e93967a5951456933, State: Initialized, Role: FOLLOWER
I20260620 11:42:57.571719 15092 ts_tablet_manager.cc:1428] T ffcee0b5b0e648df9c9ad247ad77c93e P d4e3cd899940498d83eba8fc5ed6b9a2: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20260620 11:42:57.571784 15091 consensus_queue.cc:260] T ffcee0b5b0e648df9c9ad247ad77c93e P 82db96227ab34c6e93967a5951456933 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "580c848b417d4ffbab6d865896c13de7" member_type: VOTER last_known_addr { host: "127.6.234.193" port: 44461 } } peers { permanent_uuid: "82db96227ab34c6e93967a5951456933" member_type: VOTER last_known_addr { host: "127.6.234.194" port: 40195 } } peers { permanent_uuid: "d4e3cd899940498d83eba8fc5ed6b9a2" member_type: VOTER last_known_addr { host: "127.6.234.195" port: 38309 } }
I20260620 11:42:57.571837 15066 heartbeater.cc:503] Master 127.6.234.254:44159 was elected leader, sending a full tablet report...
I20260620 11:42:57.572038 15091 ts_tablet_manager.cc:1428] T ffcee0b5b0e648df9c9ad247ad77c93e P 82db96227ab34c6e93967a5951456933: Time spent starting tablet: real 0.003s user 0.003s sys 0.000s
I20260620 11:42:57.572077 14926 heartbeater.cc:503] Master 127.6.234.254:44159 was elected leader, sending a full tablet report...
I20260620 11:42:57.572576 15093 raft_consensus.cc:348] T ffcee0b5b0e648df9c9ad247ad77c93e P 580c848b417d4ffbab6d865896c13de7 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "580c848b417d4ffbab6d865896c13de7" member_type: VOTER last_known_addr { host: "127.6.234.193" port: 44461 } } peers { permanent_uuid: "82db96227ab34c6e93967a5951456933" member_type: VOTER last_known_addr { host: "127.6.234.194" port: 40195 } } peers { permanent_uuid: "d4e3cd899940498d83eba8fc5ed6b9a2" member_type: VOTER last_known_addr { host: "127.6.234.195" port: 38309 } }
I20260620 11:42:57.572865 15093 raft_consensus.cc:374] T ffcee0b5b0e648df9c9ad247ad77c93e P 580c848b417d4ffbab6d865896c13de7 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 11:42:57.572918 15093 raft_consensus.cc:729] T ffcee0b5b0e648df9c9ad247ad77c93e P 580c848b417d4ffbab6d865896c13de7 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 580c848b417d4ffbab6d865896c13de7, State: Initialized, Role: FOLLOWER
I20260620 11:42:57.573081 15093 consensus_queue.cc:260] T ffcee0b5b0e648df9c9ad247ad77c93e P 580c848b417d4ffbab6d865896c13de7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "580c848b417d4ffbab6d865896c13de7" member_type: VOTER last_known_addr { host: "127.6.234.193" port: 44461 } } peers { permanent_uuid: "82db96227ab34c6e93967a5951456933" member_type: VOTER last_known_addr { host: "127.6.234.194" port: 40195 } } peers { permanent_uuid: "d4e3cd899940498d83eba8fc5ed6b9a2" member_type: VOTER last_known_addr { host: "127.6.234.195" port: 38309 } }
I20260620 11:42:57.573346 15093 ts_tablet_manager.cc:1428] T ffcee0b5b0e648df9c9ad247ad77c93e P 580c848b417d4ffbab6d865896c13de7: Time spent starting tablet: real 0.003s user 0.003s sys 0.000s
I20260620 11:42:57.573379 14786 heartbeater.cc:503] Master 127.6.234.254:44159 was elected leader, sending a full tablet report...
W20260620 11:42:57.622076 14927 tablet.cc:2369] T ffcee0b5b0e648df9c9ad247ad77c93e P 82db96227ab34c6e93967a5951456933: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260620 11:42:57.657519 15097 raft_consensus.cc:482] T ffcee0b5b0e648df9c9ad247ad77c93e P d4e3cd899940498d83eba8fc5ed6b9a2 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260620 11:42:57.657637 15097 raft_consensus.cc:504] T ffcee0b5b0e648df9c9ad247ad77c93e P d4e3cd899940498d83eba8fc5ed6b9a2 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "580c848b417d4ffbab6d865896c13de7" member_type: VOTER last_known_addr { host: "127.6.234.193" port: 44461 } } peers { permanent_uuid: "82db96227ab34c6e93967a5951456933" member_type: VOTER last_known_addr { host: "127.6.234.194" port: 40195 } } peers { permanent_uuid: "d4e3cd899940498d83eba8fc5ed6b9a2" member_type: VOTER last_known_addr { host: "127.6.234.195" port: 38309 } }
I20260620 11:42:57.658032 15097 leader_election.cc:288] T ffcee0b5b0e648df9c9ad247ad77c93e P d4e3cd899940498d83eba8fc5ed6b9a2 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 580c848b417d4ffbab6d865896c13de7 (127.6.234.193:44461), 82db96227ab34c6e93967a5951456933 (127.6.234.194:40195)
I20260620 11:42:57.660917 15089 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 11:42:57.658356 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.6.234.195:38489 (local address 127.6.234.193:44461)
0620 11:42:57.658405 (+ 49us) server_negotiation.cc:206] Beginning negotiation
0620 11:42:57.658407 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 11:42:57.658417 (+ 10us) server_negotiation.cc:407] Connection header received
0620 11:42:57.658448 (+ 31us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 11:42:57.658448 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 11:42:57.658459 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 11:42:57.658518 (+ 59us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 11:42:57.658715 (+ 197us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 11:42:57.659222 (+ 507us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 11:42:57.660173 (+ 951us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 11:42:57.660538 (+ 365us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 11:42:57.660593 (+ 55us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 11:42:57.660601 (+ 8us) server_negotiation.cc:1035] Waiting for connection context
0620 11:42:57.660804 (+ 203us) server_negotiation.cc:299] Negotiation successful
0620 11:42:57.660840 (+ 36us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":25}
I20260620 11:42:57.660919 15088 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 11:42:57.658362 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.6.234.195:49067 (local address 127.6.234.194:40195)
0620 11:42:57.658409 (+ 47us) server_negotiation.cc:206] Beginning negotiation
0620 11:42:57.658411 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 11:42:57.658524 (+ 113us) server_negotiation.cc:407] Connection header received
0620 11:42:57.658552 (+ 28us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 11:42:57.658553 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 11:42:57.658561 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 11:42:57.658619 (+ 58us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 11:42:57.658820 (+ 201us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 11:42:57.659283 (+ 463us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 11:42:57.660212 (+ 929us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 11:42:57.660541 (+ 329us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 11:42:57.660643 (+ 102us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 11:42:57.660653 (+ 10us) server_negotiation.cc:1035] Waiting for connection context
0620 11:42:57.660812 (+ 159us) server_negotiation.cc:299] Negotiation successful
0620 11:42:57.660846 (+ 34us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":24}
I20260620 11:42:57.660936 14959 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 11:42:57.658210 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.234.193:44461 (local address 127.6.234.195:38489)
0620 11:42:57.658260 (+ 50us) negotiation.cc:107] Waiting for socket to connect
0620 11:42:57.658271 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0620 11:42:57.658345 (+ 74us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 11:42:57.658539 (+ 194us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 11:42:57.658540 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 11:42:57.658542 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 11:42:57.658619 (+ 77us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 11:42:57.658621 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 11:42:57.659301 (+ 680us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 11:42:57.659302 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 11:42:57.660078 (+ 776us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 11:42:57.660080 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 11:42:57.660621 (+ 541us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 11:42:57.660622 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 11:42:57.660719 (+ 97us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 11:42:57.660723 (+ 4us) client_negotiation.cc:769] Sending connection context
0620 11:42:57.660798 (+ 75us) client_negotiation.cc:240] Negotiation successful
0620 11:42:57.660840 (+ 42us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":28}
I20260620 11:42:57.660939 15100 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 11:42:57.658247 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.234.194:40195 (local address 127.6.234.195:49067)
0620 11:42:57.658436 (+ 189us) negotiation.cc:107] Waiting for socket to connect
0620 11:42:57.658446 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0620 11:42:57.658507 (+ 61us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 11:42:57.658651 (+ 144us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 11:42:57.658651 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 11:42:57.658653 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 11:42:57.658749 (+ 96us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 11:42:57.658750 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 11:42:57.659371 (+ 621us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 11:42:57.659372 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 11:42:57.660091 (+ 719us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 11:42:57.660093 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 11:42:57.660662 (+ 569us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 11:42:57.660663 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 11:42:57.660731 (+ 68us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 11:42:57.660734 (+ 3us) client_negotiation.cc:769] Sending connection context
0620 11:42:57.660804 (+ 70us) client_negotiation.cc:240] Negotiation successful
0620 11:42:57.660850 (+ 46us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":173,"mutex_wait_us":3,"thread_start_us":108,"threads_started":1}
I20260620 11:42:57.661319 14740 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ffcee0b5b0e648df9c9ad247ad77c93e" candidate_uuid: "d4e3cd899940498d83eba8fc5ed6b9a2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "580c848b417d4ffbab6d865896c13de7" is_pre_election: true
I20260620 11:42:57.661396 14880 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ffcee0b5b0e648df9c9ad247ad77c93e" candidate_uuid: "d4e3cd899940498d83eba8fc5ed6b9a2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "82db96227ab34c6e93967a5951456933" is_pre_election: true
I20260620 11:42:57.661499 14740 raft_consensus.cc:2454] T ffcee0b5b0e648df9c9ad247ad77c93e P 580c848b417d4ffbab6d865896c13de7 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d4e3cd899940498d83eba8fc5ed6b9a2 in term 0.
I20260620 11:42:57.661549 14880 raft_consensus.cc:2454] T ffcee0b5b0e648df9c9ad247ad77c93e P 82db96227ab34c6e93967a5951456933 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d4e3cd899940498d83eba8fc5ed6b9a2 in term 0.
I20260620 11:42:57.661741 14951 leader_election.cc:302] T ffcee0b5b0e648df9c9ad247ad77c93e P d4e3cd899940498d83eba8fc5ed6b9a2 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 580c848b417d4ffbab6d865896c13de7, d4e3cd899940498d83eba8fc5ed6b9a2; no voters:
I20260620 11:42:57.661888 15097 raft_consensus.cc:2781] T ffcee0b5b0e648df9c9ad247ad77c93e P d4e3cd899940498d83eba8fc5ed6b9a2 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260620 11:42:57.661957 15097 raft_consensus.cc:482] T ffcee0b5b0e648df9c9ad247ad77c93e P d4e3cd899940498d83eba8fc5ed6b9a2 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260620 11:42:57.661993 15097 raft_consensus.cc:3037] T ffcee0b5b0e648df9c9ad247ad77c93e P d4e3cd899940498d83eba8fc5ed6b9a2 [term 0 FOLLOWER]: Advancing to term 1
I20260620 11:42:57.662879 15097 raft_consensus.cc:504] T ffcee0b5b0e648df9c9ad247ad77c93e P d4e3cd899940498d83eba8fc5ed6b9a2 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "580c848b417d4ffbab6d865896c13de7" member_type: VOTER last_known_addr { host: "127.6.234.193" port: 44461 } } peers { permanent_uuid: "82db96227ab34c6e93967a5951456933" member_type: VOTER last_known_addr { host: "127.6.234.194" port: 40195 } } peers { permanent_uuid: "d4e3cd899940498d83eba8fc5ed6b9a2" member_type: VOTER last_known_addr { host: "127.6.234.195" port: 38309 } }
I20260620 11:42:57.663025 15097 leader_election.cc:288] T ffcee0b5b0e648df9c9ad247ad77c93e P d4e3cd899940498d83eba8fc5ed6b9a2 [CANDIDATE]: Term 1 election: Requested vote from peers 580c848b417d4ffbab6d865896c13de7 (127.6.234.193:44461), 82db96227ab34c6e93967a5951456933 (127.6.234.194:40195)
I20260620 11:42:57.663158 14740 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ffcee0b5b0e648df9c9ad247ad77c93e" candidate_uuid: "d4e3cd899940498d83eba8fc5ed6b9a2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "580c848b417d4ffbab6d865896c13de7"
I20260620 11:42:57.663213 14880 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ffcee0b5b0e648df9c9ad247ad77c93e" candidate_uuid: "d4e3cd899940498d83eba8fc5ed6b9a2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "82db96227ab34c6e93967a5951456933"
I20260620 11:42:57.663239 14740 raft_consensus.cc:3037] T ffcee0b5b0e648df9c9ad247ad77c93e P 580c848b417d4ffbab6d865896c13de7 [term 0 FOLLOWER]: Advancing to term 1
I20260620 11:42:57.663313 14880 raft_consensus.cc:3037] T ffcee0b5b0e648df9c9ad247ad77c93e P 82db96227ab34c6e93967a5951456933 [term 0 FOLLOWER]: Advancing to term 1
I20260620 11:42:57.663947 14740 raft_consensus.cc:2454] T ffcee0b5b0e648df9c9ad247ad77c93e P 580c848b417d4ffbab6d865896c13de7 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d4e3cd899940498d83eba8fc5ed6b9a2 in term 1.
I20260620 11:42:57.663964 14880 raft_consensus.cc:2454] T ffcee0b5b0e648df9c9ad247ad77c93e P 82db96227ab34c6e93967a5951456933 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d4e3cd899940498d83eba8fc5ed6b9a2 in term 1.
I20260620 11:42:57.664130 14951 leader_election.cc:302] T ffcee0b5b0e648df9c9ad247ad77c93e P d4e3cd899940498d83eba8fc5ed6b9a2 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 580c848b417d4ffbab6d865896c13de7, d4e3cd899940498d83eba8fc5ed6b9a2; no voters:
I20260620 11:42:57.664244 15097 raft_consensus.cc:2781] T ffcee0b5b0e648df9c9ad247ad77c93e P d4e3cd899940498d83eba8fc5ed6b9a2 [term 1 FOLLOWER]: Leader election won for term 1
I20260620 11:42:57.664425 15097 raft_consensus.cc:686] T ffcee0b5b0e648df9c9ad247ad77c93e P d4e3cd899940498d83eba8fc5ed6b9a2 [term 1 LEADER]: Becoming Leader. State: Replica: d4e3cd899940498d83eba8fc5ed6b9a2, State: Running, Role: LEADER
I20260620 11:42:57.664531 15097 consensus_queue.cc:237] T ffcee0b5b0e648df9c9ad247ad77c93e P d4e3cd899940498d83eba8fc5ed6b9a2 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "580c848b417d4ffbab6d865896c13de7" member_type: VOTER last_known_addr { host: "127.6.234.193" port: 44461 } } peers { permanent_uuid: "82db96227ab34c6e93967a5951456933" member_type: VOTER last_known_addr { host: "127.6.234.194" port: 40195 } } peers { permanent_uuid: "d4e3cd899940498d83eba8fc5ed6b9a2" member_type: VOTER last_known_addr { host: "127.6.234.195" port: 38309 } }
I20260620 11:42:57.665405 14581 catalog_manager.cc:5495] T ffcee0b5b0e648df9c9ad247ad77c93e P d4e3cd899940498d83eba8fc5ed6b9a2 reported cstate change: term changed from 0 to 1, leader changed from <none> to d4e3cd899940498d83eba8fc5ed6b9a2 (127.6.234.195). New cstate: current_term: 1 leader_uuid: "d4e3cd899940498d83eba8fc5ed6b9a2" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "580c848b417d4ffbab6d865896c13de7" member_type: VOTER last_known_addr { host: "127.6.234.193" port: 44461 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "82db96227ab34c6e93967a5951456933" member_type: VOTER last_known_addr { host: "127.6.234.194" port: 40195 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "d4e3cd899940498d83eba8fc5ed6b9a2" member_type: VOTER last_known_addr { host: "127.6.234.195" port: 38309 } health_report { overall_health: HEALTHY } } }
I20260620 11:42:57.679342 15090 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 11:42:57.677296 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44466 (local address 127.6.234.195:38309)
0620 11:42:57.677356 (+ 60us) server_negotiation.cc:206] Beginning negotiation
0620 11:42:57.677358 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 11:42:57.677366 (+ 8us) server_negotiation.cc:407] Connection header received
0620 11:42:57.677408 (+ 42us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 11:42:57.677409 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 11:42:57.677423 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 11:42:57.677473 (+ 50us) server_negotiation.cc:226] Negotiated authn=TOKEN
0620 11:42:57.677563 (+ 90us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 11:42:57.678024 (+ 461us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 11:42:57.678714 (+ 690us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 11:42:57.678887 (+ 173us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 11:42:57.678934 (+ 47us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 11:42:57.679061 (+ 127us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0620 11:42:57.679148 (+ 87us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0620 11:42:57.679204 (+ 56us) server_negotiation.cc:1035] Waiting for connection context
0620 11:42:57.679237 (+ 33us) server_negotiation.cc:299] Negotiation successful
0620 11:42:57.679252 (+ 15us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":41}
I20260620 11:42:57.681037 14880 raft_consensus.cc:1261] T ffcee0b5b0e648df9c9ad247ad77c93e P 82db96227ab34c6e93967a5951456933 [term 1 FOLLOWER]: Refusing update from remote peer d4e3cd899940498d83eba8fc5ed6b9a2: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 11:42:57.681084 14740 raft_consensus.cc:1261] T ffcee0b5b0e648df9c9ad247ad77c93e P 580c848b417d4ffbab6d865896c13de7 [term 1 FOLLOWER]: Refusing update from remote peer d4e3cd899940498d83eba8fc5ed6b9a2: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 11:42:57.681365 15097 consensus_queue.cc:1035] T ffcee0b5b0e648df9c9ad247ad77c93e P d4e3cd899940498d83eba8fc5ed6b9a2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "82db96227ab34c6e93967a5951456933" member_type: VOTER last_known_addr { host: "127.6.234.194" port: 40195 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 11:42:57.681509 15101 consensus_queue.cc:1035] T ffcee0b5b0e648df9c9ad247ad77c93e P d4e3cd899940498d83eba8fc5ed6b9a2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "580c848b417d4ffbab6d865896c13de7" member_type: VOTER last_known_addr { host: "127.6.234.193" port: 44461 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 11:42:57.686082 14583 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:46544:
table { table_name: "test-table" } modify_external_catalogs: true
I20260620 11:42:57.686180 14583 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:46544:
table { table_name: "test-table" } modify_external_catalogs: true
I20260620 11:42:57.686904 14583 catalog_manager.cc:5782] T 00000000000000000000000000000000 P f75e5ca955994ed9ba7f8b6ce11b9c5a: Sending DeleteTablet for 3 replicas of tablet ffcee0b5b0e648df9c9ad247ad77c93e
I20260620 11:42:57.687283 14860 tablet_service.cc:1514] Processing DeleteTablet for tablet ffcee0b5b0e648df9c9ad247ad77c93e with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 11:42:57 UTC) from {username='oryx', principal='oryx/127.6.234.254@KRBTEST.COM'} at 127.0.0.1:39656
I20260620 11:42:57.687283 15000 tablet_service.cc:1514] Processing DeleteTablet for tablet ffcee0b5b0e648df9c9ad247ad77c93e with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 11:42:57 UTC) from {username='oryx', principal='oryx/127.6.234.254@KRBTEST.COM'} at 127.0.0.1:44454
I20260620 11:42:57.687297 14720 tablet_service.cc:1514] Processing DeleteTablet for tablet ffcee0b5b0e648df9c9ad247ad77c93e with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 11:42:57 UTC) from {username='oryx', principal='oryx/127.6.234.254@KRBTEST.COM'} at 127.0.0.1:40862
I20260620 11:42:57.687623 15115 tablet_replica.cc:330] T ffcee0b5b0e648df9c9ad247ad77c93e P 580c848b417d4ffbab6d865896c13de7: stopping tablet replica
I20260620 11:42:57.687718 15115 raft_consensus.cc:2229] T ffcee0b5b0e648df9c9ad247ad77c93e P 580c848b417d4ffbab6d865896c13de7 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260620 11:42:57.687706 14583 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:46544:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20260620 11:42:57.687820 15115 raft_consensus.cc:2258] T ffcee0b5b0e648df9c9ad247ad77c93e P 580c848b417d4ffbab6d865896c13de7 [term 1 FOLLOWER]: Raft consensus is shut down!
W20260620 11:42:57.687824 14583 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260620 11:42:57.687968 15113 tablet_replica.cc:330] T ffcee0b5b0e648df9c9ad247ad77c93e P 82db96227ab34c6e93967a5951456933: stopping tablet replica
I20260620 11:42:57.688047 15113 raft_consensus.cc:2229] T ffcee0b5b0e648df9c9ad247ad77c93e P 82db96227ab34c6e93967a5951456933 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260620 11:42:57.688768 15115 ts_tablet_manager.cc:1905] T ffcee0b5b0e648df9c9ad247ad77c93e P 580c848b417d4ffbab6d865896c13de7: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 11:42:57.688984 14720 tablet_service.cc:1467] Processing CreateTablet for tablet 9c4f8a06010d4327ac4bfdd2993d4a16 (DEFAULT_TABLE table=test-table [id=f799b00c4a34402db2ff1ac2e5a8350d]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 11:42:57.689038 15000 tablet_service.cc:1467] Processing CreateTablet for tablet 9c4f8a06010d4327ac4bfdd2993d4a16 (DEFAULT_TABLE table=test-table [id=f799b00c4a34402db2ff1ac2e5a8350d]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 11:42:57.689100 14720 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 9c4f8a06010d4327ac4bfdd2993d4a16. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 11:42:57.689118 14860 tablet_service.cc:1467] Processing CreateTablet for tablet 9c4f8a06010d4327ac4bfdd2993d4a16 (DEFAULT_TABLE table=test-table [id=f799b00c4a34402db2ff1ac2e5a8350d]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 11:42:57.689154 15000 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 9c4f8a06010d4327ac4bfdd2993d4a16. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 11:42:57.689211 14860 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 9c4f8a06010d4327ac4bfdd2993d4a16. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 11:42:57.690502 15091 tablet_bootstrap.cc:492] T 9c4f8a06010d4327ac4bfdd2993d4a16 P 82db96227ab34c6e93967a5951456933: Bootstrap starting.
I20260620 11:42:57.690517 15092 tablet_bootstrap.cc:492] T 9c4f8a06010d4327ac4bfdd2993d4a16 P d4e3cd899940498d83eba8fc5ed6b9a2: Bootstrap starting.
I20260620 11:42:57.690665 15093 tablet_bootstrap.cc:492] T 9c4f8a06010d4327ac4bfdd2993d4a16 P 580c848b417d4ffbab6d865896c13de7: Bootstrap starting.
I20260620 11:42:57.690943 15092 tablet_bootstrap.cc:654] T 9c4f8a06010d4327ac4bfdd2993d4a16 P d4e3cd899940498d83eba8fc5ed6b9a2: Neither blocks nor log segments found. Creating new log.
I20260620 11:42:57.690958 15091 tablet_bootstrap.cc:654] T 9c4f8a06010d4327ac4bfdd2993d4a16 P 82db96227ab34c6e93967a5951456933: Neither blocks nor log segments found. Creating new log.
I20260620 11:42:57.691043 15093 tablet_bootstrap.cc:654] T 9c4f8a06010d4327ac4bfdd2993d4a16 P 580c848b417d4ffbab6d865896c13de7: Neither blocks nor log segments found. Creating new log.
I20260620 11:42:57.691603 15092 tablet_bootstrap.cc:492] T 9c4f8a06010d4327ac4bfdd2993d4a16 P d4e3cd899940498d83eba8fc5ed6b9a2: No bootstrap required, opened a new log
I20260620 11:42:57.691603 15093 tablet_bootstrap.cc:492] T 9c4f8a06010d4327ac4bfdd2993d4a16 P 580c848b417d4ffbab6d865896c13de7: No bootstrap required, opened a new log
I20260620 11:42:57.691650 15093 ts_tablet_manager.cc:1397] T 9c4f8a06010d4327ac4bfdd2993d4a16 P 580c848b417d4ffbab6d865896c13de7: Time spent bootstrapping tablet: real 0.001s user 0.001s sys 0.000s
I20260620 11:42:57.691650 15092 ts_tablet_manager.cc:1397] T 9c4f8a06010d4327ac4bfdd2993d4a16 P d4e3cd899940498d83eba8fc5ed6b9a2: Time spent bootstrapping tablet: real 0.001s user 0.001s sys 0.000s
I20260620 11:42:57.691792 15092 raft_consensus.cc:348] T 9c4f8a06010d4327ac4bfdd2993d4a16 P d4e3cd899940498d83eba8fc5ed6b9a2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "580c848b417d4ffbab6d865896c13de7" member_type: VOTER last_known_addr { host: "127.6.234.193" port: 44461 } } peers { permanent_uuid: "82db96227ab34c6e93967a5951456933" member_type: VOTER last_known_addr { host: "127.6.234.194" port: 40195 } } peers { permanent_uuid: "d4e3cd899940498d83eba8fc5ed6b9a2" member_type: VOTER last_known_addr { host: "127.6.234.195" port: 38309 } }
I20260620 11:42:57.691792 15093 raft_consensus.cc:348] T 9c4f8a06010d4327ac4bfdd2993d4a16 P 580c848b417d4ffbab6d865896c13de7 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "580c848b417d4ffbab6d865896c13de7" member_type: VOTER last_known_addr { host: "127.6.234.193" port: 44461 } } peers { permanent_uuid: "82db96227ab34c6e93967a5951456933" member_type: VOTER last_known_addr { host: "127.6.234.194" port: 40195 } } peers { permanent_uuid: "d4e3cd899940498d83eba8fc5ed6b9a2" member_type: VOTER last_known_addr { host: "127.6.234.195" port: 38309 } }
I20260620 11:42:57.691852 15092 raft_consensus.cc:374] T 9c4f8a06010d4327ac4bfdd2993d4a16 P d4e3cd899940498d83eba8fc5ed6b9a2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 11:42:57.691874 15092 raft_consensus.cc:729] T 9c4f8a06010d4327ac4bfdd2993d4a16 P d4e3cd899940498d83eba8fc5ed6b9a2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d4e3cd899940498d83eba8fc5ed6b9a2, State: Initialized, Role: FOLLOWER
I20260620 11:42:57.691874 15093 raft_consensus.cc:374] T 9c4f8a06010d4327ac4bfdd2993d4a16 P 580c848b417d4ffbab6d865896c13de7 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 11:42:57.691895 15093 raft_consensus.cc:729] T 9c4f8a06010d4327ac4bfdd2993d4a16 P 580c848b417d4ffbab6d865896c13de7 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 580c848b417d4ffbab6d865896c13de7, State: Initialized, Role: FOLLOWER
I20260620 11:42:57.691917 15092 consensus_queue.cc:260] T 9c4f8a06010d4327ac4bfdd2993d4a16 P d4e3cd899940498d83eba8fc5ed6b9a2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "580c848b417d4ffbab6d865896c13de7" member_type: VOTER last_known_addr { host: "127.6.234.193" port: 44461 } } peers { permanent_uuid: "82db96227ab34c6e93967a5951456933" member_type: VOTER last_known_addr { host: "127.6.234.194" port: 40195 } } peers { permanent_uuid: "d4e3cd899940498d83eba8fc5ed6b9a2" member_type: VOTER last_known_addr { host: "127.6.234.195" port: 38309 } }
I20260620 11:42:57.691933 15093 consensus_queue.cc:260] T 9c4f8a06010d4327ac4bfdd2993d4a16 P 580c848b417d4ffbab6d865896c13de7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "580c848b417d4ffbab6d865896c13de7" member_type: VOTER last_known_addr { host: "127.6.234.193" port: 44461 } } peers { permanent_uuid: "82db96227ab34c6e93967a5951456933" member_type: VOTER last_known_addr { host: "127.6.234.194" port: 40195 } } peers { permanent_uuid: "d4e3cd899940498d83eba8fc5ed6b9a2" member_type: VOTER last_known_addr { host: "127.6.234.195" port: 38309 } }
I20260620 11:42:57.692006 15092 ts_tablet_manager.cc:1428] T 9c4f8a06010d4327ac4bfdd2993d4a16 P d4e3cd899940498d83eba8fc5ed6b9a2: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
I20260620 11:42:57.692013 15093 ts_tablet_manager.cc:1428] T 9c4f8a06010d4327ac4bfdd2993d4a16 P 580c848b417d4ffbab6d865896c13de7: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
I20260620 11:42:57.693163 15115 ts_tablet_manager.cc:1918] T ffcee0b5b0e648df9c9ad247ad77c93e P 580c848b417d4ffbab6d865896c13de7: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260620 11:42:57.693248 15115 log.cc:1196] T ffcee0b5b0e648df9c9ad247ad77c93e P 580c848b417d4ffbab6d865896c13de7: Deleting WAL directory at /tmp/dist-test-task1JRg7N/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781955746769051-7083-0/minicluster-data/ts-0/wal/wals/ffcee0b5b0e648df9c9ad247ad77c93e
I20260620 11:42:57.693244 15113 raft_consensus.cc:2258] T ffcee0b5b0e648df9c9ad247ad77c93e P 82db96227ab34c6e93967a5951456933 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 11:42:57.693622 15115 ts_tablet_manager.cc:1939] T ffcee0b5b0e648df9c9ad247ad77c93e P 580c848b417d4ffbab6d865896c13de7: Deleting consensus metadata
I20260620 11:42:57.693887 15113 ts_tablet_manager.cc:1905] T ffcee0b5b0e648df9c9ad247ad77c93e P 82db96227ab34c6e93967a5951456933: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 11:42:57.694054 14567 catalog_manager.cc:4841] TS 580c848b417d4ffbab6d865896c13de7 (127.6.234.193:44461): tablet ffcee0b5b0e648df9c9ad247ad77c93e (table test-table [id=9c7e9a98c7f8401c8edc7e99004e722b]) successfully deleted
I20260620 11:42:57.695274 15113 ts_tablet_manager.cc:1918] T ffcee0b5b0e648df9c9ad247ad77c93e P 82db96227ab34c6e93967a5951456933: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260620 11:42:57.695346 15113 log.cc:1196] T ffcee0b5b0e648df9c9ad247ad77c93e P 82db96227ab34c6e93967a5951456933: Deleting WAL directory at /tmp/dist-test-task1JRg7N/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781955746769051-7083-0/minicluster-data/ts-1/wal/wals/ffcee0b5b0e648df9c9ad247ad77c93e
I20260620 11:42:57.695616 15113 ts_tablet_manager.cc:1939] T ffcee0b5b0e648df9c9ad247ad77c93e P 82db96227ab34c6e93967a5951456933: Deleting consensus metadata
I20260620 11:42:57.696003 14570 catalog_manager.cc:4841] TS 82db96227ab34c6e93967a5951456933 (127.6.234.194:40195): tablet ffcee0b5b0e648df9c9ad247ad77c93e (table test-table [id=9c7e9a98c7f8401c8edc7e99004e722b]) successfully deleted
I20260620 11:42:57.696159 15114 tablet_replica.cc:330] T ffcee0b5b0e648df9c9ad247ad77c93e P d4e3cd899940498d83eba8fc5ed6b9a2: stopping tablet replica
I20260620 11:42:57.696252 15114 raft_consensus.cc:2229] T ffcee0b5b0e648df9c9ad247ad77c93e P d4e3cd899940498d83eba8fc5ed6b9a2 [term 1 LEADER]: Raft consensus shutting down.
I20260620 11:42:57.696341 15114 raft_consensus.cc:2258] T ffcee0b5b0e648df9c9ad247ad77c93e P d4e3cd899940498d83eba8fc5ed6b9a2 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 11:42:57.696692 15114 ts_tablet_manager.cc:1905] T ffcee0b5b0e648df9c9ad247ad77c93e P d4e3cd899940498d83eba8fc5ed6b9a2: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 11:42:57.698083 15091 tablet_bootstrap.cc:492] T 9c4f8a06010d4327ac4bfdd2993d4a16 P 82db96227ab34c6e93967a5951456933: No bootstrap required, opened a new log
I20260620 11:42:57.698143 15114 ts_tablet_manager.cc:1918] T ffcee0b5b0e648df9c9ad247ad77c93e P d4e3cd899940498d83eba8fc5ed6b9a2: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260620 11:42:57.698158 15091 ts_tablet_manager.cc:1397] T 9c4f8a06010d4327ac4bfdd2993d4a16 P 82db96227ab34c6e93967a5951456933: Time spent bootstrapping tablet: real 0.008s user 0.001s sys 0.000s
I20260620 11:42:57.698210 15114 log.cc:1196] T ffcee0b5b0e648df9c9ad247ad77c93e P d4e3cd899940498d83eba8fc5ed6b9a2: Deleting WAL directory at /tmp/dist-test-task1JRg7N/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781955746769051-7083-0/minicluster-data/ts-2/wal/wals/ffcee0b5b0e648df9c9ad247ad77c93e
I20260620 11:42:57.698304 15091 raft_consensus.cc:348] T 9c4f8a06010d4327ac4bfdd2993d4a16 P 82db96227ab34c6e93967a5951456933 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "580c848b417d4ffbab6d865896c13de7" member_type: VOTER last_known_addr { host: "127.6.234.193" port: 44461 } } peers { permanent_uuid: "82db96227ab34c6e93967a5951456933" member_type: VOTER last_known_addr { host: "127.6.234.194" port: 40195 } } peers { permanent_uuid: "d4e3cd899940498d83eba8fc5ed6b9a2" member_type: VOTER last_known_addr { host: "127.6.234.195" port: 38309 } }
I20260620 11:42:57.698392 15091 raft_consensus.cc:374] T 9c4f8a06010d4327ac4bfdd2993d4a16 P 82db96227ab34c6e93967a5951456933 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 11:42:57.698419 15091 raft_consensus.cc:729] T 9c4f8a06010d4327ac4bfdd2993d4a16 P 82db96227ab34c6e93967a5951456933 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 82db96227ab34c6e93967a5951456933, State: Initialized, Role: FOLLOWER
I20260620 11:42:57.698477 15091 consensus_queue.cc:260] T 9c4f8a06010d4327ac4bfdd2993d4a16 P 82db96227ab34c6e93967a5951456933 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "580c848b417d4ffbab6d865896c13de7" member_type: VOTER last_known_addr { host: "127.6.234.193" port: 44461 } } peers { permanent_uuid: "82db96227ab34c6e93967a5951456933" member_type: VOTER last_known_addr { host: "127.6.234.194" port: 40195 } } peers { permanent_uuid: "d4e3cd899940498d83eba8fc5ed6b9a2" member_type: VOTER last_known_addr { host: "127.6.234.195" port: 38309 } }
I20260620 11:42:57.698498 15114 ts_tablet_manager.cc:1939] T ffcee0b5b0e648df9c9ad247ad77c93e P d4e3cd899940498d83eba8fc5ed6b9a2: Deleting consensus metadata
I20260620 11:42:57.698571 15091 ts_tablet_manager.cc:1428] T 9c4f8a06010d4327ac4bfdd2993d4a16 P 82db96227ab34c6e93967a5951456933: Time spent starting tablet: real 0.000s user 0.001s sys 0.000s
I20260620 11:42:57.699018 14569 catalog_manager.cc:4841] TS d4e3cd899940498d83eba8fc5ed6b9a2 (127.6.234.195:38309): tablet ffcee0b5b0e648df9c9ad247ad77c93e (table test-table [id=9c7e9a98c7f8401c8edc7e99004e722b]) successfully deleted
W20260620 11:42:57.720356 14787 tablet.cc:2369] T 9c4f8a06010d4327ac4bfdd2993d4a16 P 580c848b417d4ffbab6d865896c13de7: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260620 11:42:57.746954 15098 raft_consensus.cc:482] T 9c4f8a06010d4327ac4bfdd2993d4a16 P 82db96227ab34c6e93967a5951456933 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260620 11:42:57.747085 15098 raft_consensus.cc:504] T 9c4f8a06010d4327ac4bfdd2993d4a16 P 82db96227ab34c6e93967a5951456933 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "580c848b417d4ffbab6d865896c13de7" member_type: VOTER last_known_addr { host: "127.6.234.193" port: 44461 } } peers { permanent_uuid: "82db96227ab34c6e93967a5951456933" member_type: VOTER last_known_addr { host: "127.6.234.194" port: 40195 } } peers { permanent_uuid: "d4e3cd899940498d83eba8fc5ed6b9a2" member_type: VOTER last_known_addr { host: "127.6.234.195" port: 38309 } }
I20260620 11:42:57.747449 15098 leader_election.cc:288] T 9c4f8a06010d4327ac4bfdd2993d4a16 P 82db96227ab34c6e93967a5951456933 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 580c848b417d4ffbab6d865896c13de7 (127.6.234.193:44461), d4e3cd899940498d83eba8fc5ed6b9a2 (127.6.234.195:38309)
I20260620 11:42:57.751008 15089 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 11:42:57.747722 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.6.234.194:56535 (local address 127.6.234.193:44461)
0620 11:42:57.747771 (+ 49us) server_negotiation.cc:206] Beginning negotiation
0620 11:42:57.747773 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 11:42:57.747850 (+ 77us) server_negotiation.cc:407] Connection header received
0620 11:42:57.747928 (+ 78us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 11:42:57.747930 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 11:42:57.747939 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 11:42:57.747991 (+ 52us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 11:42:57.748275 (+ 284us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 11:42:57.748979 (+ 704us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 11:42:57.750300 (+ 1321us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 11:42:57.750647 (+ 347us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 11:42:57.750744 (+ 97us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 11:42:57.750753 (+ 9us) server_negotiation.cc:1035] Waiting for connection context
0620 11:42:57.750900 (+ 147us) server_negotiation.cc:299] Negotiation successful
0620 11:42:57.750939 (+ 39us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":24}
I20260620 11:42:57.751009 14819 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 11:42:57.747662 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.234.193:44461 (local address 127.6.234.194:56535)
0620 11:42:57.747761 (+ 99us) negotiation.cc:107] Waiting for socket to connect
0620 11:42:57.747773 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0620 11:42:57.747837 (+ 64us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 11:42:57.748129 (+ 292us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 11:42:57.748130 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 11:42:57.748131 (+ 1us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 11:42:57.748191 (+ 60us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 11:42:57.748193 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 11:42:57.749064 (+ 871us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 11:42:57.749064 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 11:42:57.750181 (+ 1117us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 11:42:57.750184 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 11:42:57.750744 (+ 560us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 11:42:57.750745 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 11:42:57.750824 (+ 79us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 11:42:57.750827 (+ 3us) client_negotiation.cc:769] Sending connection context
0620 11:42:57.750895 (+ 68us) client_negotiation.cc:240] Negotiation successful
0620 11:42:57.750941 (+ 46us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":76,"mutex_wait_us":2}
I20260620 11:42:57.751197 15090 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 11:42:57.747932 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.6.234.194:51505 (local address 127.6.234.195:38309)
0620 11:42:57.747980 (+ 48us) server_negotiation.cc:206] Beginning negotiation
0620 11:42:57.747982 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 11:42:57.747990 (+ 8us) server_negotiation.cc:407] Connection header received
0620 11:42:57.748014 (+ 24us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 11:42:57.748015 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 11:42:57.748023 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 11:42:57.748081 (+ 58us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 11:42:57.748264 (+ 183us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 11:42:57.748737 (+ 473us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 11:42:57.750267 (+ 1530us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 11:42:57.750821 (+ 554us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 11:42:57.750892 (+ 71us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 11:42:57.750900 (+ 8us) server_negotiation.cc:1035] Waiting for connection context
0620 11:42:57.751095 (+ 195us) server_negotiation.cc:299] Negotiation successful
0620 11:42:57.751133 (+ 38us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":24}
I20260620 11:42:57.751442 15119 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 11:42:57.747663 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.234.195:38309 (local address 127.6.234.194:51505)
0620 11:42:57.747861 (+ 198us) negotiation.cc:107] Waiting for socket to connect
0620 11:42:57.747871 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0620 11:42:57.747924 (+ 53us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 11:42:57.748086 (+ 162us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 11:42:57.748087 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 11:42:57.748089 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 11:42:57.748171 (+ 82us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 11:42:57.748173 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 11:42:57.748883 (+ 710us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 11:42:57.748884 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 11:42:57.750181 (+ 1297us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 11:42:57.750184 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 11:42:57.750935 (+ 751us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 11:42:57.750936 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 11:42:57.751014 (+ 78us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 11:42:57.751018 (+ 4us) client_negotiation.cc:769] Sending connection context
0620 11:42:57.751309 (+ 291us) client_negotiation.cc:240] Negotiation successful
0620 11:42:57.751350 (+ 41us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":184,"mutex_wait_us":57,"thread_start_us":61,"threads_started":1}
I20260620 11:42:57.751597 14740 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9c4f8a06010d4327ac4bfdd2993d4a16" candidate_uuid: "82db96227ab34c6e93967a5951456933" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "580c848b417d4ffbab6d865896c13de7" is_pre_election: true
I20260620 11:42:57.751694 14740 raft_consensus.cc:2454] T 9c4f8a06010d4327ac4bfdd2993d4a16 P 580c848b417d4ffbab6d865896c13de7 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 82db96227ab34c6e93967a5951456933 in term 0.
I20260620 11:42:57.751677 15020 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9c4f8a06010d4327ac4bfdd2993d4a16" candidate_uuid: "82db96227ab34c6e93967a5951456933" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d4e3cd899940498d83eba8fc5ed6b9a2" is_pre_election: true
I20260620 11:42:57.751783 15020 raft_consensus.cc:2454] T 9c4f8a06010d4327ac4bfdd2993d4a16 P d4e3cd899940498d83eba8fc5ed6b9a2 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 82db96227ab34c6e93967a5951456933 in term 0.
I20260620 11:42:57.751879 14811 leader_election.cc:302] T 9c4f8a06010d4327ac4bfdd2993d4a16 P 82db96227ab34c6e93967a5951456933 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 580c848b417d4ffbab6d865896c13de7, 82db96227ab34c6e93967a5951456933; no voters:
I20260620 11:42:57.752029 15098 raft_consensus.cc:2781] T 9c4f8a06010d4327ac4bfdd2993d4a16 P 82db96227ab34c6e93967a5951456933 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260620 11:42:57.752066 15098 raft_consensus.cc:482] T 9c4f8a06010d4327ac4bfdd2993d4a16 P 82db96227ab34c6e93967a5951456933 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260620 11:42:57.752091 15098 raft_consensus.cc:3037] T 9c4f8a06010d4327ac4bfdd2993d4a16 P 82db96227ab34c6e93967a5951456933 [term 0 FOLLOWER]: Advancing to term 1
I20260620 11:42:57.752964 15098 raft_consensus.cc:504] T 9c4f8a06010d4327ac4bfdd2993d4a16 P 82db96227ab34c6e93967a5951456933 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "580c848b417d4ffbab6d865896c13de7" member_type: VOTER last_known_addr { host: "127.6.234.193" port: 44461 } } peers { permanent_uuid: "82db96227ab34c6e93967a5951456933" member_type: VOTER last_known_addr { host: "127.6.234.194" port: 40195 } } peers { permanent_uuid: "d4e3cd899940498d83eba8fc5ed6b9a2" member_type: VOTER last_known_addr { host: "127.6.234.195" port: 38309 } }
I20260620 11:42:57.753100 15098 leader_election.cc:288] T 9c4f8a06010d4327ac4bfdd2993d4a16 P 82db96227ab34c6e93967a5951456933 [CANDIDATE]: Term 1 election: Requested vote from peers 580c848b417d4ffbab6d865896c13de7 (127.6.234.193:44461), d4e3cd899940498d83eba8fc5ed6b9a2 (127.6.234.195:38309)
I20260620 11:42:57.753233 14740 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9c4f8a06010d4327ac4bfdd2993d4a16" candidate_uuid: "82db96227ab34c6e93967a5951456933" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "580c848b417d4ffbab6d865896c13de7"
I20260620 11:42:57.753302 14740 raft_consensus.cc:3037] T 9c4f8a06010d4327ac4bfdd2993d4a16 P 580c848b417d4ffbab6d865896c13de7 [term 0 FOLLOWER]: Advancing to term 1
I20260620 11:42:57.753275 15020 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9c4f8a06010d4327ac4bfdd2993d4a16" candidate_uuid: "82db96227ab34c6e93967a5951456933" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d4e3cd899940498d83eba8fc5ed6b9a2"
I20260620 11:42:57.753350 15020 raft_consensus.cc:3037] T 9c4f8a06010d4327ac4bfdd2993d4a16 P d4e3cd899940498d83eba8fc5ed6b9a2 [term 0 FOLLOWER]: Advancing to term 1
I20260620 11:42:57.754060 14740 raft_consensus.cc:2454] T 9c4f8a06010d4327ac4bfdd2993d4a16 P 580c848b417d4ffbab6d865896c13de7 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 82db96227ab34c6e93967a5951456933 in term 1.
I20260620 11:42:57.754226 14811 leader_election.cc:302] T 9c4f8a06010d4327ac4bfdd2993d4a16 P 82db96227ab34c6e93967a5951456933 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 580c848b417d4ffbab6d865896c13de7, 82db96227ab34c6e93967a5951456933; no voters:
I20260620 11:42:57.754268 15020 raft_consensus.cc:2454] T 9c4f8a06010d4327ac4bfdd2993d4a16 P d4e3cd899940498d83eba8fc5ed6b9a2 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 82db96227ab34c6e93967a5951456933 in term 1.
I20260620 11:42:57.754307 15098 raft_consensus.cc:2781] T 9c4f8a06010d4327ac4bfdd2993d4a16 P 82db96227ab34c6e93967a5951456933 [term 1 FOLLOWER]: Leader election won for term 1
I20260620 11:42:57.754416 15098 raft_consensus.cc:686] T 9c4f8a06010d4327ac4bfdd2993d4a16 P 82db96227ab34c6e93967a5951456933 [term 1 LEADER]: Becoming Leader. State: Replica: 82db96227ab34c6e93967a5951456933, State: Running, Role: LEADER
I20260620 11:42:57.754487 15098 consensus_queue.cc:237] T 9c4f8a06010d4327ac4bfdd2993d4a16 P 82db96227ab34c6e93967a5951456933 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "580c848b417d4ffbab6d865896c13de7" member_type: VOTER last_known_addr { host: "127.6.234.193" port: 44461 } } peers { permanent_uuid: "82db96227ab34c6e93967a5951456933" member_type: VOTER last_known_addr { host: "127.6.234.194" port: 40195 } } peers { permanent_uuid: "d4e3cd899940498d83eba8fc5ed6b9a2" member_type: VOTER last_known_addr { host: "127.6.234.195" port: 38309 } }
I20260620 11:42:57.755177 14583 catalog_manager.cc:5495] T 9c4f8a06010d4327ac4bfdd2993d4a16 P 82db96227ab34c6e93967a5951456933 reported cstate change: term changed from 0 to 1, leader changed from <none> to 82db96227ab34c6e93967a5951456933 (127.6.234.194). New cstate: current_term: 1 leader_uuid: "82db96227ab34c6e93967a5951456933" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "580c848b417d4ffbab6d865896c13de7" member_type: VOTER last_known_addr { host: "127.6.234.193" port: 44461 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "82db96227ab34c6e93967a5951456933" member_type: VOTER last_known_addr { host: "127.6.234.194" port: 40195 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "d4e3cd899940498d83eba8fc5ed6b9a2" member_type: VOTER last_known_addr { host: "127.6.234.195" port: 38309 } health_report { overall_health: UNKNOWN } } }
W20260620 11:42:57.766206 15067 tablet.cc:2369] T 9c4f8a06010d4327ac4bfdd2993d4a16 P d4e3cd899940498d83eba8fc5ed6b9a2: Can't schedule compaction. Clean time has not been advanced past its initial value.
Jun 20 11:42:57 dist-test-slave-cjc6 krb5kdc[14538](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781955776, etypes {rep=17 tkt=17 ses=17}, oryx/127.6.234.254@KRBTEST.COM for oryx/127.6.234.254@KRBTEST.COM
I20260620 11:42:57.799304 14633 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 11:42:57.790563 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46550 (local address 127.6.234.254:44159)
0620 11:42:57.792310 (+ 1747us) server_negotiation.cc:206] Beginning negotiation
0620 11:42:57.792312 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 11:42:57.792330 (+ 18us) server_negotiation.cc:407] Connection header received
0620 11:42:57.792355 (+ 25us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 11:42:57.792356 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 11:42:57.792374 (+ 18us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 11:42:57.792447 (+ 73us) server_negotiation.cc:226] Negotiated authn=SASL
0620 11:42:57.792924 (+ 477us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 11:42:57.793443 (+ 519us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 11:42:57.794220 (+ 777us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 11:42:57.794428 (+ 208us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 11:42:57.794503 (+ 75us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 11:42:57.796773 (+ 2270us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 11:42:57.796775 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 11:42:57.796775 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 11:42:57.796782 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0620 11:42:57.798125 (+ 1343us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 11:42:57.798511 (+ 386us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 11:42:57.798512 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 11:42:57.798513 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 11:42:57.798565 (+ 52us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 11:42:57.798767 (+ 202us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 11:42:57.798768 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 11:42:57.798768 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 11:42:57.798884 (+ 116us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 11:42:57.798959 (+ 75us) server_negotiation.cc:1035] Waiting for connection context
0620 11:42:57.799112 (+ 153us) server_negotiation.cc:299] Negotiation successful
0620 11:42:57.799192 (+ 80us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1719}
I20260620 11:42:57.799633 15127 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 11:42:57.790541 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.234.254:44159 (local address 127.0.0.1:46550)
0620 11:42:57.790687 (+ 146us) negotiation.cc:107] Waiting for socket to connect
0620 11:42:57.790698 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0620 11:42:57.790761 (+ 63us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 11:42:57.792454 (+ 1693us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 11:42:57.792455 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 11:42:57.792766 (+ 311us) client_negotiation.cc:189] Negotiated authn=SASL
0620 11:42:57.792812 (+ 46us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 11:42:57.792814 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 11:42:57.793528 (+ 714us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 11:42:57.793529 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 11:42:57.794134 (+ 605us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 11:42:57.794136 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 11:42:57.794512 (+ 376us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 11:42:57.794513 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 11:42:57.794573 (+ 60us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 11:42:57.795153 (+ 580us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 11:42:57.795158 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0620 11:42:57.796685 (+ 1527us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 11:42:57.798223 (+ 1538us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 11:42:57.798231 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 11:42:57.798233 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0620 11:42:57.798426 (+ 193us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 11:42:57.798628 (+ 202us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 11:42:57.798629 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 11:42:57.798629 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0620 11:42:57.798704 (+ 75us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 11:42:57.798949 (+ 245us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 11:42:57.798956 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 11:42:57.799001 (+ 45us) client_negotiation.cc:769] Sending connection context
0620 11:42:57.799090 (+ 89us) client_negotiation.cc:240] Negotiation successful
0620 11:42:57.799248 (+ 158us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":115,"mutex_wait_us":59,"thread_start_us":89,"threads_started":1}
I20260620 11:42:57.801754 14583 catalog_manager.cc:1908] Servicing CreateTable request from {username='oryx', principal='oryx/127.6.234.254@KRBTEST.COM'} at 127.0.0.1:46550:
name: "kudu_system.kudu_transactions"
schema {
columns {
name: "txn_id"
type: INT64
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "entry_type"
type: INT8
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "identifier"
type: STRING
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "metadata"
type: STRING
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
rows: "<redacted>""\006\001\000\000\000\000\000\000\000\000\007\001@B\017\000\000\000\000\000"
indirect_data: "<redacted>"""
}
partition_schema {
range_schema {
columns {
name: "txn_id"
}
}
}
table_type: TXN_STATUS_TABLE
W20260620 11:42:57.801987 14583 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table kudu_system.kudu_transactions in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260620 11:42:57.803314 14720 tablet_service.cc:1467] Processing CreateTablet for tablet 5dccd8071bbf4a75acc960c9b3bdab30 (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=efdc7994fc4745b89da90d87a23cd12a]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260620 11:42:57.803356 15000 tablet_service.cc:1467] Processing CreateTablet for tablet 5dccd8071bbf4a75acc960c9b3bdab30 (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=efdc7994fc4745b89da90d87a23cd12a]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260620 11:42:57.803475 14720 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 5dccd8071bbf4a75acc960c9b3bdab30. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 11:42:57.803505 15000 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 5dccd8071bbf4a75acc960c9b3bdab30. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 11:42:57.804548 15092 tablet_bootstrap.cc:492] T 5dccd8071bbf4a75acc960c9b3bdab30 P d4e3cd899940498d83eba8fc5ed6b9a2: Bootstrap starting.
I20260620 11:42:57.804651 14860 tablet_service.cc:1467] Processing CreateTablet for tablet 5dccd8071bbf4a75acc960c9b3bdab30 (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=efdc7994fc4745b89da90d87a23cd12a]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260620 11:42:57.804778 14860 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 5dccd8071bbf4a75acc960c9b3bdab30. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 11:42:57.804862 15092 tablet_bootstrap.cc:654] T 5dccd8071bbf4a75acc960c9b3bdab30 P d4e3cd899940498d83eba8fc5ed6b9a2: Neither blocks nor log segments found. Creating new log.
I20260620 11:42:57.805105 15093 tablet_bootstrap.cc:492] T 5dccd8071bbf4a75acc960c9b3bdab30 P 580c848b417d4ffbab6d865896c13de7: Bootstrap starting.
I20260620 11:42:57.805493 15093 tablet_bootstrap.cc:654] T 5dccd8071bbf4a75acc960c9b3bdab30 P 580c848b417d4ffbab6d865896c13de7: Neither blocks nor log segments found. Creating new log.
I20260620 11:42:57.806273 15093 tablet_bootstrap.cc:492] T 5dccd8071bbf4a75acc960c9b3bdab30 P 580c848b417d4ffbab6d865896c13de7: No bootstrap required, opened a new log
I20260620 11:42:57.806295 15092 tablet_bootstrap.cc:492] T 5dccd8071bbf4a75acc960c9b3bdab30 P d4e3cd899940498d83eba8fc5ed6b9a2: No bootstrap required, opened a new log
I20260620 11:42:57.806321 15093 ts_tablet_manager.cc:1397] T 5dccd8071bbf4a75acc960c9b3bdab30 P 580c848b417d4ffbab6d865896c13de7: Time spent bootstrapping tablet: real 0.001s user 0.000s sys 0.001s
I20260620 11:42:57.806337 15092 ts_tablet_manager.cc:1397] T 5dccd8071bbf4a75acc960c9b3bdab30 P d4e3cd899940498d83eba8fc5ed6b9a2: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260620 11:42:57.806455 15092 raft_consensus.cc:348] T 5dccd8071bbf4a75acc960c9b3bdab30 P d4e3cd899940498d83eba8fc5ed6b9a2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "580c848b417d4ffbab6d865896c13de7" member_type: VOTER last_known_addr { host: "127.6.234.193" port: 44461 } } peers { permanent_uuid: "82db96227ab34c6e93967a5951456933" member_type: VOTER last_known_addr { host: "127.6.234.194" port: 40195 } } peers { permanent_uuid: "d4e3cd899940498d83eba8fc5ed6b9a2" member_type: VOTER last_known_addr { host: "127.6.234.195" port: 38309 } }
I20260620 11:42:57.806457 15093 raft_consensus.cc:348] T 5dccd8071bbf4a75acc960c9b3bdab30 P 580c848b417d4ffbab6d865896c13de7 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "580c848b417d4ffbab6d865896c13de7" member_type: VOTER last_known_addr { host: "127.6.234.193" port: 44461 } } peers { permanent_uuid: "82db96227ab34c6e93967a5951456933" member_type: VOTER last_known_addr { host: "127.6.234.194" port: 40195 } } peers { permanent_uuid: "d4e3cd899940498d83eba8fc5ed6b9a2" member_type: VOTER last_known_addr { host: "127.6.234.195" port: 38309 } }
I20260620 11:42:57.806520 15092 raft_consensus.cc:374] T 5dccd8071bbf4a75acc960c9b3bdab30 P d4e3cd899940498d83eba8fc5ed6b9a2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 11:42:57.806524 15093 raft_consensus.cc:374] T 5dccd8071bbf4a75acc960c9b3bdab30 P 580c848b417d4ffbab6d865896c13de7 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 11:42:57.806517 15091 tablet_bootstrap.cc:492] T 5dccd8071bbf4a75acc960c9b3bdab30 P 82db96227ab34c6e93967a5951456933: Bootstrap starting.
I20260620 11:42:57.806540 15092 raft_consensus.cc:729] T 5dccd8071bbf4a75acc960c9b3bdab30 P d4e3cd899940498d83eba8fc5ed6b9a2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d4e3cd899940498d83eba8fc5ed6b9a2, State: Initialized, Role: FOLLOWER
I20260620 11:42:57.806545 15093 raft_consensus.cc:729] T 5dccd8071bbf4a75acc960c9b3bdab30 P 580c848b417d4ffbab6d865896c13de7 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 580c848b417d4ffbab6d865896c13de7, State: Initialized, Role: FOLLOWER
I20260620 11:42:57.806578 15092 consensus_queue.cc:260] T 5dccd8071bbf4a75acc960c9b3bdab30 P d4e3cd899940498d83eba8fc5ed6b9a2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "580c848b417d4ffbab6d865896c13de7" member_type: VOTER last_known_addr { host: "127.6.234.193" port: 44461 } } peers { permanent_uuid: "82db96227ab34c6e93967a5951456933" member_type: VOTER last_known_addr { host: "127.6.234.194" port: 40195 } } peers { permanent_uuid: "d4e3cd899940498d83eba8fc5ed6b9a2" member_type: VOTER last_known_addr { host: "127.6.234.195" port: 38309 } }
I20260620 11:42:57.806586 15093 consensus_queue.cc:260] T 5dccd8071bbf4a75acc960c9b3bdab30 P 580c848b417d4ffbab6d865896c13de7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "580c848b417d4ffbab6d865896c13de7" member_type: VOTER last_known_addr { host: "127.6.234.193" port: 44461 } } peers { permanent_uuid: "82db96227ab34c6e93967a5951456933" member_type: VOTER last_known_addr { host: "127.6.234.194" port: 40195 } } peers { permanent_uuid: "d4e3cd899940498d83eba8fc5ed6b9a2" member_type: VOTER last_known_addr { host: "127.6.234.195" port: 38309 } }
I20260620 11:42:57.806658 15092 ts_tablet_manager.cc:1428] T 5dccd8071bbf4a75acc960c9b3bdab30 P d4e3cd899940498d83eba8fc5ed6b9a2: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
I20260620 11:42:57.806682 15093 ts_tablet_manager.cc:1428] T 5dccd8071bbf4a75acc960c9b3bdab30 P 580c848b417d4ffbab6d865896c13de7: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
I20260620 11:42:57.806914 15091 tablet_bootstrap.cc:654] T 5dccd8071bbf4a75acc960c9b3bdab30 P 82db96227ab34c6e93967a5951456933: Neither blocks nor log segments found. Creating new log.
I20260620 11:42:57.807389 15101 tablet_replica.cc:439] T 5dccd8071bbf4a75acc960c9b3bdab30 P d4e3cd899940498d83eba8fc5ed6b9a2: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "580c848b417d4ffbab6d865896c13de7" member_type: VOTER last_known_addr { host: "127.6.234.193" port: 44461 } } peers { permanent_uuid: "82db96227ab34c6e93967a5951456933" member_type: VOTER last_known_addr { host: "127.6.234.194" port: 40195 } } peers { permanent_uuid: "d4e3cd899940498d83eba8fc5ed6b9a2" member_type: VOTER last_known_addr { host: "127.6.234.195" port: 38309 } } }
I20260620 11:42:57.807505 15101 tablet_replica.cc:442] T 5dccd8071bbf4a75acc960c9b3bdab30 P d4e3cd899940498d83eba8fc5ed6b9a2: This TxnStatusTablet replica's current role is: FOLLOWER
I20260620 11:42:57.807631 15099 tablet_replica.cc:439] T 5dccd8071bbf4a75acc960c9b3bdab30 P 580c848b417d4ffbab6d865896c13de7: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "580c848b417d4ffbab6d865896c13de7" member_type: VOTER last_known_addr { host: "127.6.234.193" port: 44461 } } peers { permanent_uuid: "82db96227ab34c6e93967a5951456933" member_type: VOTER last_known_addr { host: "127.6.234.194" port: 40195 } } peers { permanent_uuid: "d4e3cd899940498d83eba8fc5ed6b9a2" member_type: VOTER last_known_addr { host: "127.6.234.195" port: 38309 } } }
I20260620 11:42:57.807688 15099 tablet_replica.cc:442] T 5dccd8071bbf4a75acc960c9b3bdab30 P 580c848b417d4ffbab6d865896c13de7: This TxnStatusTablet replica's current role is: FOLLOWER
I20260620 11:42:57.808200 15091 tablet_bootstrap.cc:492] T 5dccd8071bbf4a75acc960c9b3bdab30 P 82db96227ab34c6e93967a5951456933: No bootstrap required, opened a new log
I20260620 11:42:57.808267 15091 ts_tablet_manager.cc:1397] T 5dccd8071bbf4a75acc960c9b3bdab30 P 82db96227ab34c6e93967a5951456933: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260620 11:42:57.808421 15091 raft_consensus.cc:348] T 5dccd8071bbf4a75acc960c9b3bdab30 P 82db96227ab34c6e93967a5951456933 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "580c848b417d4ffbab6d865896c13de7" member_type: VOTER last_known_addr { host: "127.6.234.193" port: 44461 } } peers { permanent_uuid: "82db96227ab34c6e93967a5951456933" member_type: VOTER last_known_addr { host: "127.6.234.194" port: 40195 } } peers { permanent_uuid: "d4e3cd899940498d83eba8fc5ed6b9a2" member_type: VOTER last_known_addr { host: "127.6.234.195" port: 38309 } }
I20260620 11:42:57.808491 15091 raft_consensus.cc:374] T 5dccd8071bbf4a75acc960c9b3bdab30 P 82db96227ab34c6e93967a5951456933 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 11:42:57.808526 15091 raft_consensus.cc:729] T 5dccd8071bbf4a75acc960c9b3bdab30 P 82db96227ab34c6e93967a5951456933 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 82db96227ab34c6e93967a5951456933, State: Initialized, Role: FOLLOWER
I20260620 11:42:57.808574 15091 consensus_queue.cc:260] T 5dccd8071bbf4a75acc960c9b3bdab30 P 82db96227ab34c6e93967a5951456933 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "580c848b417d4ffbab6d865896c13de7" member_type: VOTER last_known_addr { host: "127.6.234.193" port: 44461 } } peers { permanent_uuid: "82db96227ab34c6e93967a5951456933" member_type: VOTER last_known_addr { host: "127.6.234.194" port: 40195 } } peers { permanent_uuid: "d4e3cd899940498d83eba8fc5ed6b9a2" member_type: VOTER last_known_addr { host: "127.6.234.195" port: 38309 } }
I20260620 11:42:57.808660 15091 ts_tablet_manager.cc:1428] T 5dccd8071bbf4a75acc960c9b3bdab30 P 82db96227ab34c6e93967a5951456933: Time spent starting tablet: real 0.000s user 0.001s sys 0.000s
I20260620 11:42:57.808785 15098 tablet_replica.cc:439] T 5dccd8071bbf4a75acc960c9b3bdab30 P 82db96227ab34c6e93967a5951456933: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "580c848b417d4ffbab6d865896c13de7" member_type: VOTER last_known_addr { host: "127.6.234.193" port: 44461 } } peers { permanent_uuid: "82db96227ab34c6e93967a5951456933" member_type: VOTER last_known_addr { host: "127.6.234.194" port: 40195 } } peers { permanent_uuid: "d4e3cd899940498d83eba8fc5ed6b9a2" member_type: VOTER last_known_addr { host: "127.6.234.195" port: 38309 } } }
I20260620 11:42:57.808881 15098 tablet_replica.cc:442] T 5dccd8071bbf4a75acc960c9b3bdab30 P 82db96227ab34c6e93967a5951456933: This TxnStatusTablet replica's current role is: FOLLOWER
I20260620 11:42:57.826946 15098 raft_consensus.cc:482] T 5dccd8071bbf4a75acc960c9b3bdab30 P 82db96227ab34c6e93967a5951456933 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260620 11:42:57.827011 15098 raft_consensus.cc:504] T 5dccd8071bbf4a75acc960c9b3bdab30 P 82db96227ab34c6e93967a5951456933 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "580c848b417d4ffbab6d865896c13de7" member_type: VOTER last_known_addr { host: "127.6.234.193" port: 44461 } } peers { permanent_uuid: "82db96227ab34c6e93967a5951456933" member_type: VOTER last_known_addr { host: "127.6.234.194" port: 40195 } } peers { permanent_uuid: "d4e3cd899940498d83eba8fc5ed6b9a2" member_type: VOTER last_known_addr { host: "127.6.234.195" port: 38309 } }
I20260620 11:42:57.827140 15098 leader_election.cc:288] T 5dccd8071bbf4a75acc960c9b3bdab30 P 82db96227ab34c6e93967a5951456933 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 580c848b417d4ffbab6d865896c13de7 (127.6.234.193:44461), d4e3cd899940498d83eba8fc5ed6b9a2 (127.6.234.195:38309)
I20260620 11:42:57.827296 14740 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5dccd8071bbf4a75acc960c9b3bdab30" candidate_uuid: "82db96227ab34c6e93967a5951456933" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "580c848b417d4ffbab6d865896c13de7" is_pre_election: true
I20260620 11:42:57.827332 15020 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5dccd8071bbf4a75acc960c9b3bdab30" candidate_uuid: "82db96227ab34c6e93967a5951456933" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d4e3cd899940498d83eba8fc5ed6b9a2" is_pre_election: true
I20260620 11:42:57.827397 14740 raft_consensus.cc:2454] T 5dccd8071bbf4a75acc960c9b3bdab30 P 580c848b417d4ffbab6d865896c13de7 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 82db96227ab34c6e93967a5951456933 in term 0.
I20260620 11:42:57.827406 15020 raft_consensus.cc:2454] T 5dccd8071bbf4a75acc960c9b3bdab30 P d4e3cd899940498d83eba8fc5ed6b9a2 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 82db96227ab34c6e93967a5951456933 in term 0.
I20260620 11:42:57.827526 14811 leader_election.cc:302] T 5dccd8071bbf4a75acc960c9b3bdab30 P 82db96227ab34c6e93967a5951456933 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 580c848b417d4ffbab6d865896c13de7, 82db96227ab34c6e93967a5951456933; no voters:
I20260620 11:42:57.827632 15098 raft_consensus.cc:2781] T 5dccd8071bbf4a75acc960c9b3bdab30 P 82db96227ab34c6e93967a5951456933 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260620 11:42:57.827672 15098 raft_consensus.cc:482] T 5dccd8071bbf4a75acc960c9b3bdab30 P 82db96227ab34c6e93967a5951456933 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260620 11:42:57.827701 15098 raft_consensus.cc:3037] T 5dccd8071bbf4a75acc960c9b3bdab30 P 82db96227ab34c6e93967a5951456933 [term 0 FOLLOWER]: Advancing to term 1
I20260620 11:42:57.828583 15098 raft_consensus.cc:504] T 5dccd8071bbf4a75acc960c9b3bdab30 P 82db96227ab34c6e93967a5951456933 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "580c848b417d4ffbab6d865896c13de7" member_type: VOTER last_known_addr { host: "127.6.234.193" port: 44461 } } peers { permanent_uuid: "82db96227ab34c6e93967a5951456933" member_type: VOTER last_known_addr { host: "127.6.234.194" port: 40195 } } peers { permanent_uuid: "d4e3cd899940498d83eba8fc5ed6b9a2" member_type: VOTER last_known_addr { host: "127.6.234.195" port: 38309 } }
I20260620 11:42:57.828722 15098 leader_election.cc:288] T 5dccd8071bbf4a75acc960c9b3bdab30 P 82db96227ab34c6e93967a5951456933 [CANDIDATE]: Term 1 election: Requested vote from peers 580c848b417d4ffbab6d865896c13de7 (127.6.234.193:44461), d4e3cd899940498d83eba8fc5ed6b9a2 (127.6.234.195:38309)
I20260620 11:42:57.828867 14740 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5dccd8071bbf4a75acc960c9b3bdab30" candidate_uuid: "82db96227ab34c6e93967a5951456933" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "580c848b417d4ffbab6d865896c13de7"
I20260620 11:42:57.828941 14740 raft_consensus.cc:3037] T 5dccd8071bbf4a75acc960c9b3bdab30 P 580c848b417d4ffbab6d865896c13de7 [term 0 FOLLOWER]: Advancing to term 1
I20260620 11:42:57.828914 15020 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5dccd8071bbf4a75acc960c9b3bdab30" candidate_uuid: "82db96227ab34c6e93967a5951456933" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d4e3cd899940498d83eba8fc5ed6b9a2"
I20260620 11:42:57.828998 15020 raft_consensus.cc:3037] T 5dccd8071bbf4a75acc960c9b3bdab30 P d4e3cd899940498d83eba8fc5ed6b9a2 [term 0 FOLLOWER]: Advancing to term 1
I20260620 11:42:57.829456 14740 raft_consensus.cc:2454] T 5dccd8071bbf4a75acc960c9b3bdab30 P 580c848b417d4ffbab6d865896c13de7 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 82db96227ab34c6e93967a5951456933 in term 1.
I20260620 11:42:57.829627 15020 raft_consensus.cc:2454] T 5dccd8071bbf4a75acc960c9b3bdab30 P d4e3cd899940498d83eba8fc5ed6b9a2 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 82db96227ab34c6e93967a5951456933 in term 1.
I20260620 11:42:57.829631 14811 leader_election.cc:302] T 5dccd8071bbf4a75acc960c9b3bdab30 P 82db96227ab34c6e93967a5951456933 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 580c848b417d4ffbab6d865896c13de7, 82db96227ab34c6e93967a5951456933; no voters:
I20260620 11:42:57.829766 15098 raft_consensus.cc:2781] T 5dccd8071bbf4a75acc960c9b3bdab30 P 82db96227ab34c6e93967a5951456933 [term 1 FOLLOWER]: Leader election won for term 1
I20260620 11:42:57.829828 15098 raft_consensus.cc:686] T 5dccd8071bbf4a75acc960c9b3bdab30 P 82db96227ab34c6e93967a5951456933 [term 1 LEADER]: Becoming Leader. State: Replica: 82db96227ab34c6e93967a5951456933, State: Running, Role: LEADER
I20260620 11:42:57.829887 15098 consensus_queue.cc:237] T 5dccd8071bbf4a75acc960c9b3bdab30 P 82db96227ab34c6e93967a5951456933 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "580c848b417d4ffbab6d865896c13de7" member_type: VOTER last_known_addr { host: "127.6.234.193" port: 44461 } } peers { permanent_uuid: "82db96227ab34c6e93967a5951456933" member_type: VOTER last_known_addr { host: "127.6.234.194" port: 40195 } } peers { permanent_uuid: "d4e3cd899940498d83eba8fc5ed6b9a2" member_type: VOTER last_known_addr { host: "127.6.234.195" port: 38309 } }
I20260620 11:42:57.830175 15120 tablet_replica.cc:439] T 5dccd8071bbf4a75acc960c9b3bdab30 P 82db96227ab34c6e93967a5951456933: TxnStatusTablet state changed. Reason: New leader 82db96227ab34c6e93967a5951456933. Latest consensus state: current_term: 1 leader_uuid: "82db96227ab34c6e93967a5951456933" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "580c848b417d4ffbab6d865896c13de7" member_type: VOTER last_known_addr { host: "127.6.234.193" port: 44461 } } peers { permanent_uuid: "82db96227ab34c6e93967a5951456933" member_type: VOTER last_known_addr { host: "127.6.234.194" port: 40195 } } peers { permanent_uuid: "d4e3cd899940498d83eba8fc5ed6b9a2" member_type: VOTER last_known_addr { host: "127.6.234.195" port: 38309 } } }
I20260620 11:42:57.830260 15120 tablet_replica.cc:442] T 5dccd8071bbf4a75acc960c9b3bdab30 P 82db96227ab34c6e93967a5951456933: This TxnStatusTablet replica's current role is: LEADER
I20260620 11:42:57.830473 15132 txn_status_manager.cc:873] Waiting until node catch up with all replicated operations in previous term...
I20260620 11:42:57.830468 14581 catalog_manager.cc:5495] T 5dccd8071bbf4a75acc960c9b3bdab30 P 82db96227ab34c6e93967a5951456933 reported cstate change: term changed from 0 to 1, leader changed from <none> to 82db96227ab34c6e93967a5951456933 (127.6.234.194). New cstate: current_term: 1 leader_uuid: "82db96227ab34c6e93967a5951456933" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "580c848b417d4ffbab6d865896c13de7" member_type: VOTER last_known_addr { host: "127.6.234.193" port: 44461 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "82db96227ab34c6e93967a5951456933" member_type: VOTER last_known_addr { host: "127.6.234.194" port: 40195 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "d4e3cd899940498d83eba8fc5ed6b9a2" member_type: VOTER last_known_addr { host: "127.6.234.195" port: 38309 } health_report { overall_health: UNKNOWN } } }
I20260620 11:42:57.830572 15132 txn_status_manager.cc:929] Loading transaction status metadata into memory...
I20260620 11:42:57.843453 15088 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 11:42:57.840892 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39664 (local address 127.6.234.194:40195)
0620 11:42:57.840949 (+ 57us) server_negotiation.cc:206] Beginning negotiation
0620 11:42:57.840952 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0620 11:42:57.840961 (+ 9us) server_negotiation.cc:407] Connection header received
0620 11:42:57.841024 (+ 63us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 11:42:57.841025 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 11:42:57.841043 (+ 18us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 11:42:57.841105 (+ 62us) server_negotiation.cc:226] Negotiated authn=TOKEN
0620 11:42:57.841244 (+ 139us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 11:42:57.841814 (+ 570us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 11:42:57.842646 (+ 832us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 11:42:57.842856 (+ 210us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 11:42:57.842940 (+ 84us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 11:42:57.843123 (+ 183us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0620 11:42:57.843232 (+ 109us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0620 11:42:57.843313 (+ 81us) server_negotiation.cc:1035] Waiting for connection context
0620 11:42:57.843362 (+ 49us) server_negotiation.cc:299] Negotiation successful
0620 11:42:57.843382 (+ 20us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":36}
I20260620 11:42:57.843473 15127 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 11:42:57.840813 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.234.194:40195 (local address 127.0.0.1:39664)
0620 11:42:57.840888 (+ 75us) negotiation.cc:107] Waiting for socket to connect
0620 11:42:57.840900 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0620 11:42:57.840953 (+ 53us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 11:42:57.841115 (+ 162us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 11:42:57.841115 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 11:42:57.841117 (+ 2us) client_negotiation.cc:189] Negotiated authn=TOKEN
0620 11:42:57.841165 (+ 48us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 11:42:57.841168 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 11:42:57.841905 (+ 737us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 11:42:57.841906 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 11:42:57.842547 (+ 641us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 11:42:57.842549 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 11:42:57.842941 (+ 392us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 11:42:57.842942 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 11:42:57.843009 (+ 67us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 11:42:57.843032 (+ 23us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0620 11:42:57.843298 (+ 266us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0620 11:42:57.843307 (+ 9us) client_negotiation.cc:769] Sending connection context
0620 11:42:57.843356 (+ 49us) client_negotiation.cc:240] Negotiation successful
0620 11:42:57.843405 (+ 49us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":36}
I20260620 11:42:57.844522 14740 raft_consensus.cc:1261] T 5dccd8071bbf4a75acc960c9b3bdab30 P 580c848b417d4ffbab6d865896c13de7 [term 1 FOLLOWER]: Refusing update from remote peer 82db96227ab34c6e93967a5951456933: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 11:42:57.844604 15020 raft_consensus.cc:1261] T 5dccd8071bbf4a75acc960c9b3bdab30 P d4e3cd899940498d83eba8fc5ed6b9a2 [term 1 FOLLOWER]: Refusing update from remote peer 82db96227ab34c6e93967a5951456933: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 11:42:57.844767 15120 consensus_queue.cc:1035] T 5dccd8071bbf4a75acc960c9b3bdab30 P 82db96227ab34c6e93967a5951456933 [LEADER]: Connected to new peer: Peer: permanent_uuid: "580c848b417d4ffbab6d865896c13de7" member_type: VOTER last_known_addr { host: "127.6.234.193" port: 44461 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 11:42:57.844894 15098 consensus_queue.cc:1035] T 5dccd8071bbf4a75acc960c9b3bdab30 P 82db96227ab34c6e93967a5951456933 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d4e3cd899940498d83eba8fc5ed6b9a2" member_type: VOTER last_known_addr { host: "127.6.234.195" port: 38309 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 11:42:57.845556 15101 tablet_replica.cc:439] T 5dccd8071bbf4a75acc960c9b3bdab30 P d4e3cd899940498d83eba8fc5ed6b9a2: TxnStatusTablet state changed. Reason: New leader 82db96227ab34c6e93967a5951456933. Latest consensus state: current_term: 1 leader_uuid: "82db96227ab34c6e93967a5951456933" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "580c848b417d4ffbab6d865896c13de7" member_type: VOTER last_known_addr { host: "127.6.234.193" port: 44461 } } peers { permanent_uuid: "82db96227ab34c6e93967a5951456933" member_type: VOTER last_known_addr { host: "127.6.234.194" port: 40195 } } peers { permanent_uuid: "d4e3cd899940498d83eba8fc5ed6b9a2" member_type: VOTER last_known_addr { host: "127.6.234.195" port: 38309 } } }
I20260620 11:42:57.845651 15101 tablet_replica.cc:442] T 5dccd8071bbf4a75acc960c9b3bdab30 P d4e3cd899940498d83eba8fc5ed6b9a2: This TxnStatusTablet replica's current role is: FOLLOWER
I20260620 11:42:57.845759 15099 tablet_replica.cc:439] T 5dccd8071bbf4a75acc960c9b3bdab30 P 580c848b417d4ffbab6d865896c13de7: TxnStatusTablet state changed. Reason: New leader 82db96227ab34c6e93967a5951456933. Latest consensus state: current_term: 1 leader_uuid: "82db96227ab34c6e93967a5951456933" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "580c848b417d4ffbab6d865896c13de7" member_type: VOTER last_known_addr { host: "127.6.234.193" port: 44461 } } peers { permanent_uuid: "82db96227ab34c6e93967a5951456933" member_type: VOTER last_known_addr { host: "127.6.234.194" port: 40195 } } peers { permanent_uuid: "d4e3cd899940498d83eba8fc5ed6b9a2" member_type: VOTER last_known_addr { host: "127.6.234.195" port: 38309 } } }
I20260620 11:42:57.845844 15099 tablet_replica.cc:442] T 5dccd8071bbf4a75acc960c9b3bdab30 P 580c848b417d4ffbab6d865896c13de7: This TxnStatusTablet replica's current role is: FOLLOWER
I20260620 11:42:57.846211 15120 tablet_replica.cc:439] T 5dccd8071bbf4a75acc960c9b3bdab30 P 82db96227ab34c6e93967a5951456933: TxnStatusTablet state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "82db96227ab34c6e93967a5951456933" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "580c848b417d4ffbab6d865896c13de7" member_type: VOTER last_known_addr { host: "127.6.234.193" port: 44461 } } peers { permanent_uuid: "82db96227ab34c6e93967a5951456933" member_type: VOTER last_known_addr { host: "127.6.234.194" port: 40195 } } peers { permanent_uuid: "d4e3cd899940498d83eba8fc5ed6b9a2" member_type: VOTER last_known_addr { host: "127.6.234.195" port: 38309 } } }
I20260620 11:42:57.846293 15120 tablet_replica.cc:442] T 5dccd8071bbf4a75acc960c9b3bdab30 P 82db96227ab34c6e93967a5951456933: This TxnStatusTablet replica's current role is: LEADER
I20260620 11:42:57.846899 15104 mvcc.cc:205] Tried to move back new op lower bound from 7298890866049142784 to 7298890865991458816. Current Snapshot: MvccSnapshot[applied={T|T < 7298890866049142784}]
I20260620 11:42:57.846891 15120 tablet_replica.cc:439] T 5dccd8071bbf4a75acc960c9b3bdab30 P 82db96227ab34c6e93967a5951456933: TxnStatusTablet state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "82db96227ab34c6e93967a5951456933" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "580c848b417d4ffbab6d865896c13de7" member_type: VOTER last_known_addr { host: "127.6.234.193" port: 44461 } } peers { permanent_uuid: "82db96227ab34c6e93967a5951456933" member_type: VOTER last_known_addr { host: "127.6.234.194" port: 40195 } } peers { permanent_uuid: "d4e3cd899940498d83eba8fc5ed6b9a2" member_type: VOTER last_known_addr { host: "127.6.234.195" port: 38309 } } }
I20260620 11:42:57.846959 15120 tablet_replica.cc:442] T 5dccd8071bbf4a75acc960c9b3bdab30 P 82db96227ab34c6e93967a5951456933: This TxnStatusTablet replica's current role is: LEADER
I20260620 11:42:57.847033 15101 tablet_replica.cc:439] T 5dccd8071bbf4a75acc960c9b3bdab30 P d4e3cd899940498d83eba8fc5ed6b9a2: TxnStatusTablet state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "82db96227ab34c6e93967a5951456933" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "580c848b417d4ffbab6d865896c13de7" member_type: VOTER last_known_addr { host: "127.6.234.193" port: 44461 } } peers { permanent_uuid: "82db96227ab34c6e93967a5951456933" member_type: VOTER last_known_addr { host: "127.6.234.194" port: 40195 } } peers { permanent_uuid: "d4e3cd899940498d83eba8fc5ed6b9a2" member_type: VOTER last_known_addr { host: "127.6.234.195" port: 38309 } } }
I20260620 11:42:57.847105 15101 tablet_replica.cc:442] T 5dccd8071bbf4a75acc960c9b3bdab30 P d4e3cd899940498d83eba8fc5ed6b9a2: This TxnStatusTablet replica's current role is: FOLLOWER
I20260620 11:42:57.847919 15099 tablet_replica.cc:439] T 5dccd8071bbf4a75acc960c9b3bdab30 P 580c848b417d4ffbab6d865896c13de7: TxnStatusTablet state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "82db96227ab34c6e93967a5951456933" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "580c848b417d4ffbab6d865896c13de7" member_type: VOTER last_known_addr { host: "127.6.234.193" port: 44461 } } peers { permanent_uuid: "82db96227ab34c6e93967a5951456933" member_type: VOTER last_known_addr { host: "127.6.234.194" port: 40195 } } peers { permanent_uuid: "d4e3cd899940498d83eba8fc5ed6b9a2" member_type: VOTER last_known_addr { host: "127.6.234.195" port: 38309 } } }
I20260620 11:42:57.848006 15099 tablet_replica.cc:442] T 5dccd8071bbf4a75acc960c9b3bdab30 P 580c848b417d4ffbab6d865896c13de7: This TxnStatusTablet replica's current role is: FOLLOWER
I20260620 11:42:57.850693 15088 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 11:42:57.847889 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39676 (local address 127.6.234.194:40195)
0620 11:42:57.848075 (+ 186us) server_negotiation.cc:206] Beginning negotiation
0620 11:42:57.848077 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 11:42:57.848085 (+ 8us) server_negotiation.cc:407] Connection header received
0620 11:42:57.848105 (+ 20us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 11:42:57.848106 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 11:42:57.848115 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 11:42:57.848175 (+ 60us) server_negotiation.cc:226] Negotiated authn=TOKEN
0620 11:42:57.848326 (+ 151us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 11:42:57.849096 (+ 770us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 11:42:57.849927 (+ 831us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 11:42:57.850222 (+ 295us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 11:42:57.850285 (+ 63us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 11:42:57.850443 (+ 158us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0620 11:42:57.850475 (+ 32us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0620 11:42:57.850552 (+ 77us) server_negotiation.cc:1035] Waiting for connection context
0620 11:42:57.850610 (+ 58us) server_negotiation.cc:299] Negotiation successful
0620 11:42:57.850628 (+ 18us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":146}
I20260620 11:42:57.855531 15088 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 11:42:57.853097 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.6.234.194:46623 (local address 127.6.234.194:40195)
0620 11:42:57.853152 (+ 55us) server_negotiation.cc:206] Beginning negotiation
0620 11:42:57.853154 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 11:42:57.853162 (+ 8us) server_negotiation.cc:407] Connection header received
0620 11:42:57.853207 (+ 45us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 11:42:57.853208 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 11:42:57.853217 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 11:42:57.853269 (+ 52us) server_negotiation.cc:226] Negotiated authn=TOKEN
0620 11:42:57.853375 (+ 106us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 11:42:57.854087 (+ 712us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 11:42:57.854805 (+ 718us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 11:42:57.855103 (+ 298us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 11:42:57.855172 (+ 69us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 11:42:57.855303 (+ 131us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0620 11:42:57.855332 (+ 29us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0620 11:42:57.855388 (+ 56us) server_negotiation.cc:1035] Waiting for connection context
0620 11:42:57.855422 (+ 34us) server_negotiation.cc:299] Negotiation successful
0620 11:42:57.855438 (+ 16us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":37}
I20260620 11:42:57.855533 14933 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 11:42:57.853014 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.234.194:40195 (local address 127.6.234.194:46623)
0620 11:42:57.853095 (+ 81us) negotiation.cc:107] Waiting for socket to connect
0620 11:42:57.853107 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0620 11:42:57.853149 (+ 42us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 11:42:57.853272 (+ 123us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 11:42:57.853272 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 11:42:57.853273 (+ 1us) client_negotiation.cc:189] Negotiated authn=TOKEN
0620 11:42:57.853324 (+ 51us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 11:42:57.853325 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 11:42:57.854156 (+ 831us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 11:42:57.854157 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 11:42:57.854728 (+ 571us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 11:42:57.854729 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 11:42:57.855177 (+ 448us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 11:42:57.855177 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 11:42:57.855221 (+ 44us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 11:42:57.855233 (+ 12us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0620 11:42:57.855388 (+ 155us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0620 11:42:57.855389 (+ 1us) client_negotiation.cc:769] Sending connection context
0620 11:42:57.855419 (+ 30us) client_negotiation.cc:240] Negotiation successful
0620 11:42:57.855446 (+ 27us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":44}
I20260620 11:42:57.857715 14740 raft_consensus.cc:1261] T 9c4f8a06010d4327ac4bfdd2993d4a16 P 580c848b417d4ffbab6d865896c13de7 [term 1 FOLLOWER]: Refusing update from remote peer 82db96227ab34c6e93967a5951456933: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 11:42:57.857738 15020 raft_consensus.cc:1261] T 9c4f8a06010d4327ac4bfdd2993d4a16 P d4e3cd899940498d83eba8fc5ed6b9a2 [term 1 FOLLOWER]: Refusing update from remote peer 82db96227ab34c6e93967a5951456933: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 11:42:57.857951 15098 consensus_queue.cc:1035] T 9c4f8a06010d4327ac4bfdd2993d4a16 P 82db96227ab34c6e93967a5951456933 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d4e3cd899940498d83eba8fc5ed6b9a2" member_type: VOTER last_known_addr { host: "127.6.234.195" port: 38309 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 11:42:57.858055 15139 consensus_queue.cc:1035] T 9c4f8a06010d4327ac4bfdd2993d4a16 P 82db96227ab34c6e93967a5951456933 [LEADER]: Connected to new peer: Peer: permanent_uuid: "580c848b417d4ffbab6d865896c13de7" member_type: VOTER last_known_addr { host: "127.6.234.193" port: 44461 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 11:42:57.859753 15105 mvcc.cc:205] Tried to move back new op lower bound from 7298890866103697408 to 7298890865682919424. Current Snapshot: MvccSnapshot[applied={T|T < 7298890866103697408}]
I20260620 11:42:57.871539 14583 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:46544:
table { table_name: "test-table" } modify_external_catalogs: true
I20260620 11:42:57.871641 14583 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:46544:
table { table_name: "test-table" } modify_external_catalogs: true
I20260620 11:42:57.872223 14583 catalog_manager.cc:5782] T 00000000000000000000000000000000 P f75e5ca955994ed9ba7f8b6ce11b9c5a: Sending DeleteTablet for 3 replicas of tablet 9c4f8a06010d4327ac4bfdd2993d4a16
I20260620 11:42:57.872488 15000 tablet_service.cc:1514] Processing DeleteTablet for tablet 9c4f8a06010d4327ac4bfdd2993d4a16 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 11:42:57 UTC) from {username='oryx', principal='oryx/127.6.234.254@KRBTEST.COM'} at 127.0.0.1:44454
I20260620 11:42:57.872615 15114 tablet_replica.cc:330] T 9c4f8a06010d4327ac4bfdd2993d4a16 P d4e3cd899940498d83eba8fc5ed6b9a2: stopping tablet replica
I20260620 11:42:57.872646 14859 tablet_service.cc:1514] Processing DeleteTablet for tablet 9c4f8a06010d4327ac4bfdd2993d4a16 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 11:42:57 UTC) from {username='oryx', principal='oryx/127.6.234.254@KRBTEST.COM'} at 127.0.0.1:39656
I20260620 11:42:57.872696 15114 raft_consensus.cc:2229] T 9c4f8a06010d4327ac4bfdd2993d4a16 P d4e3cd899940498d83eba8fc5ed6b9a2 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260620 11:42:57.872752 15114 raft_consensus.cc:2258] T 9c4f8a06010d4327ac4bfdd2993d4a16 P d4e3cd899940498d83eba8fc5ed6b9a2 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 11:42:57.872830 14720 tablet_service.cc:1514] Processing DeleteTablet for tablet 9c4f8a06010d4327ac4bfdd2993d4a16 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 11:42:57 UTC) from {username='oryx', principal='oryx/127.6.234.254@KRBTEST.COM'} at 127.0.0.1:40862
I20260620 11:42:57.872961 15113 tablet_replica.cc:330] T 9c4f8a06010d4327ac4bfdd2993d4a16 P 82db96227ab34c6e93967a5951456933: stopping tablet replica
I20260620 11:42:57.872958 15115 tablet_replica.cc:330] T 9c4f8a06010d4327ac4bfdd2993d4a16 P 580c848b417d4ffbab6d865896c13de7: stopping tablet replica
I20260620 11:42:57.873034 15113 raft_consensus.cc:2229] T 9c4f8a06010d4327ac4bfdd2993d4a16 P 82db96227ab34c6e93967a5951456933 [term 1 LEADER]: Raft consensus shutting down.
I20260620 11:42:57.873039 15115 raft_consensus.cc:2229] T 9c4f8a06010d4327ac4bfdd2993d4a16 P 580c848b417d4ffbab6d865896c13de7 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260620 11:42:57.873075 15115 raft_consensus.cc:2258] T 9c4f8a06010d4327ac4bfdd2993d4a16 P 580c848b417d4ffbab6d865896c13de7 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 11:42:57.873121 15113 raft_consensus.cc:2258] T 9c4f8a06010d4327ac4bfdd2993d4a16 P 82db96227ab34c6e93967a5951456933 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 11:42:57.873162 15114 ts_tablet_manager.cc:1905] T 9c4f8a06010d4327ac4bfdd2993d4a16 P d4e3cd899940498d83eba8fc5ed6b9a2: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 11:42:57.873425 15113 ts_tablet_manager.cc:1905] T 9c4f8a06010d4327ac4bfdd2993d4a16 P 82db96227ab34c6e93967a5951456933: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 11:42:57.874760 15114 ts_tablet_manager.cc:1918] T 9c4f8a06010d4327ac4bfdd2993d4a16 P d4e3cd899940498d83eba8fc5ed6b9a2: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260620 11:42:57.874815 15114 log.cc:1196] T 9c4f8a06010d4327ac4bfdd2993d4a16 P d4e3cd899940498d83eba8fc5ed6b9a2: Deleting WAL directory at /tmp/dist-test-task1JRg7N/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781955746769051-7083-0/minicluster-data/ts-2/wal/wals/9c4f8a06010d4327ac4bfdd2993d4a16
I20260620 11:42:57.874925 15113 ts_tablet_manager.cc:1918] T 9c4f8a06010d4327ac4bfdd2993d4a16 P 82db96227ab34c6e93967a5951456933: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260620 11:42:57.874971 15113 log.cc:1196] T 9c4f8a06010d4327ac4bfdd2993d4a16 P 82db96227ab34c6e93967a5951456933: Deleting WAL directory at /tmp/dist-test-task1JRg7N/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781955746769051-7083-0/minicluster-data/ts-1/wal/wals/9c4f8a06010d4327ac4bfdd2993d4a16
I20260620 11:42:57.875039 15114 ts_tablet_manager.cc:1939] T 9c4f8a06010d4327ac4bfdd2993d4a16 P d4e3cd899940498d83eba8fc5ed6b9a2: Deleting consensus metadata
I20260620 11:42:57.875167 15113 ts_tablet_manager.cc:1939] T 9c4f8a06010d4327ac4bfdd2993d4a16 P 82db96227ab34c6e93967a5951456933: Deleting consensus metadata
I20260620 11:42:57.875608 14569 catalog_manager.cc:4841] TS d4e3cd899940498d83eba8fc5ed6b9a2 (127.6.234.195:38309): tablet 9c4f8a06010d4327ac4bfdd2993d4a16 (table test-table [id=f799b00c4a34402db2ff1ac2e5a8350d]) successfully deleted
I20260620 11:42:57.875795 14570 catalog_manager.cc:4841] TS 82db96227ab34c6e93967a5951456933 (127.6.234.194:40195): tablet 9c4f8a06010d4327ac4bfdd2993d4a16 (table test-table [id=f799b00c4a34402db2ff1ac2e5a8350d]) successfully deleted
I20260620 11:42:57.877445 15115 ts_tablet_manager.cc:1905] T 9c4f8a06010d4327ac4bfdd2993d4a16 P 580c848b417d4ffbab6d865896c13de7: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 11:42:57.878943 15115 ts_tablet_manager.cc:1918] T 9c4f8a06010d4327ac4bfdd2993d4a16 P 580c848b417d4ffbab6d865896c13de7: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260620 11:42:57.879009 15115 log.cc:1196] T 9c4f8a06010d4327ac4bfdd2993d4a16 P 580c848b417d4ffbab6d865896c13de7: Deleting WAL directory at /tmp/dist-test-task1JRg7N/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781955746769051-7083-0/minicluster-data/ts-0/wal/wals/9c4f8a06010d4327ac4bfdd2993d4a16
I20260620 11:42:57.879211 15115 ts_tablet_manager.cc:1939] T 9c4f8a06010d4327ac4bfdd2993d4a16 P 580c848b417d4ffbab6d865896c13de7: Deleting consensus metadata
I20260620 11:42:57.879503 14567 catalog_manager.cc:4841] TS 580c848b417d4ffbab6d865896c13de7 (127.6.234.193:44461): tablet 9c4f8a06010d4327ac4bfdd2993d4a16 (table test-table [id=f799b00c4a34402db2ff1ac2e5a8350d]) successfully deleted
Jun 20 11:42:57 dist-test-slave-cjc6 krb5kdc[14538](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781955777, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
Jun 20 11:42:57 dist-test-slave-cjc6 krb5kdc[14538](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, test-admin@KRBTEST.COM for kudu/127.6.234.254@KRBTEST.COM, Server not found in Kerberos database
Jun 20 11:42:57 dist-test-slave-cjc6 krb5kdc[14538](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, test-admin@KRBTEST.COM for kudu/127.6.234.254@KRBTEST.COM, Server not found in Kerberos database
W20260620 11:42:57.898485 14633 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 11:42:57.892862 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46554 (local address 127.6.234.254:44159)
0620 11:42:57.892939 (+ 77us) server_negotiation.cc:206] Beginning negotiation
0620 11:42:57.892942 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0620 11:42:57.893059 (+ 117us) server_negotiation.cc:407] Connection header received
0620 11:42:57.893154 (+ 95us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 11:42:57.893155 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 11:42:57.893166 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 11:42:57.893241 (+ 75us) server_negotiation.cc:226] Negotiated authn=SASL
0620 11:42:57.893883 (+ 642us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 11:42:57.894385 (+ 502us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 11:42:57.895144 (+ 759us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 11:42:57.895352 (+ 208us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 11:42:57.895431 (+ 79us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 11:42:57.898419 (+ 2988us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:46554: BlockingRecv error: recv got EOF from 127.0.0.1:46554 (error 108)
Metrics: {"server-negotiator.queue_time_us":47}
Jun 20 11:42:57 dist-test-slave-cjc6 krb5kdc[14538](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781955777, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.6.234.254@KRBTEST.COM
I20260620 11:42:57.906639 14633 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 11:42:57.899300 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46562 (local address 127.6.234.254:44159)
0620 11:42:57.899338 (+ 38us) server_negotiation.cc:206] Beginning negotiation
0620 11:42:57.899340 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 11:42:57.899394 (+ 54us) server_negotiation.cc:407] Connection header received
0620 11:42:57.899453 (+ 59us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 11:42:57.899454 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 11:42:57.899463 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 11:42:57.899526 (+ 63us) server_negotiation.cc:226] Negotiated authn=SASL
0620 11:42:57.899926 (+ 400us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 11:42:57.900589 (+ 663us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 11:42:57.901403 (+ 814us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 11:42:57.901761 (+ 358us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 11:42:57.901837 (+ 76us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 11:42:57.904017 (+ 2180us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 11:42:57.904018 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 11:42:57.904018 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 11:42:57.904026 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0620 11:42:57.905513 (+ 1487us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 11:42:57.905901 (+ 388us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 11:42:57.905902 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 11:42:57.905903 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 11:42:57.905966 (+ 63us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 11:42:57.906148 (+ 182us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 11:42:57.906149 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 11:42:57.906150 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 11:42:57.906282 (+ 132us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 11:42:57.906386 (+ 104us) server_negotiation.cc:1035] Waiting for connection context
0620 11:42:57.906467 (+ 81us) server_negotiation.cc:299] Negotiation successful
0620 11:42:57.906545 (+ 78us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":20}
I20260620 11:42:57.908509 14583 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:46562:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260620 11:42:57.908681 14583 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260620 11:42:57.910136 14720 tablet_service.cc:1467] Processing CreateTablet for tablet 11fba61e83294be4b3d4cbe731fe6c4f (DEFAULT_TABLE table=test-table [id=3fb436b92b044e269ad11ad1598cee8b]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 11:42:57.910136 14859 tablet_service.cc:1467] Processing CreateTablet for tablet 11fba61e83294be4b3d4cbe731fe6c4f (DEFAULT_TABLE table=test-table [id=3fb436b92b044e269ad11ad1598cee8b]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 11:42:57.910269 14720 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 11fba61e83294be4b3d4cbe731fe6c4f. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 11:42:57.910269 14859 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 11fba61e83294be4b3d4cbe731fe6c4f. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 11:42:57.911564 15093 tablet_bootstrap.cc:492] T 11fba61e83294be4b3d4cbe731fe6c4f P 580c848b417d4ffbab6d865896c13de7: Bootstrap starting.
I20260620 11:42:57.911576 15091 tablet_bootstrap.cc:492] T 11fba61e83294be4b3d4cbe731fe6c4f P 82db96227ab34c6e93967a5951456933: Bootstrap starting.
I20260620 11:42:57.911904 15091 tablet_bootstrap.cc:654] T 11fba61e83294be4b3d4cbe731fe6c4f P 82db96227ab34c6e93967a5951456933: Neither blocks nor log segments found. Creating new log.
I20260620 11:42:57.911929 15093 tablet_bootstrap.cc:654] T 11fba61e83294be4b3d4cbe731fe6c4f P 580c848b417d4ffbab6d865896c13de7: Neither blocks nor log segments found. Creating new log.
I20260620 11:42:57.912655 15000 tablet_service.cc:1467] Processing CreateTablet for tablet 11fba61e83294be4b3d4cbe731fe6c4f (DEFAULT_TABLE table=test-table [id=3fb436b92b044e269ad11ad1598cee8b]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 11:42:57.912779 15000 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 11fba61e83294be4b3d4cbe731fe6c4f. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 11:42:57.912792 15093 tablet_bootstrap.cc:492] T 11fba61e83294be4b3d4cbe731fe6c4f P 580c848b417d4ffbab6d865896c13de7: No bootstrap required, opened a new log
I20260620 11:42:57.912851 15093 ts_tablet_manager.cc:1397] T 11fba61e83294be4b3d4cbe731fe6c4f P 580c848b417d4ffbab6d865896c13de7: Time spent bootstrapping tablet: real 0.001s user 0.000s sys 0.001s
I20260620 11:42:57.913014 15093 raft_consensus.cc:348] T 11fba61e83294be4b3d4cbe731fe6c4f P 580c848b417d4ffbab6d865896c13de7 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "580c848b417d4ffbab6d865896c13de7" member_type: VOTER last_known_addr { host: "127.6.234.193" port: 44461 } } peers { permanent_uuid: "82db96227ab34c6e93967a5951456933" member_type: VOTER last_known_addr { host: "127.6.234.194" port: 40195 } } peers { permanent_uuid: "d4e3cd899940498d83eba8fc5ed6b9a2" member_type: VOTER last_known_addr { host: "127.6.234.195" port: 38309 } }
I20260620 11:42:57.913092 15093 raft_consensus.cc:374] T 11fba61e83294be4b3d4cbe731fe6c4f P 580c848b417d4ffbab6d865896c13de7 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 11:42:57.913125 15093 raft_consensus.cc:729] T 11fba61e83294be4b3d4cbe731fe6c4f P 580c848b417d4ffbab6d865896c13de7 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 580c848b417d4ffbab6d865896c13de7, State: Initialized, Role: FOLLOWER
I20260620 11:42:57.913165 15093 consensus_queue.cc:260] T 11fba61e83294be4b3d4cbe731fe6c4f P 580c848b417d4ffbab6d865896c13de7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "580c848b417d4ffbab6d865896c13de7" member_type: VOTER last_known_addr { host: "127.6.234.193" port: 44461 } } peers { permanent_uuid: "82db96227ab34c6e93967a5951456933" member_type: VOTER last_known_addr { host: "127.6.234.194" port: 40195 } } peers { permanent_uuid: "d4e3cd899940498d83eba8fc5ed6b9a2" member_type: VOTER last_known_addr { host: "127.6.234.195" port: 38309 } }
I20260620 11:42:57.913254 15093 ts_tablet_manager.cc:1428] T 11fba61e83294be4b3d4cbe731fe6c4f P 580c848b417d4ffbab6d865896c13de7: Time spent starting tablet: real 0.000s user 0.000s sys 0.001s
I20260620 11:42:57.913903 15091 tablet_bootstrap.cc:492] T 11fba61e83294be4b3d4cbe731fe6c4f P 82db96227ab34c6e93967a5951456933: No bootstrap required, opened a new log
I20260620 11:42:57.913986 15091 ts_tablet_manager.cc:1397] T 11fba61e83294be4b3d4cbe731fe6c4f P 82db96227ab34c6e93967a5951456933: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260620 11:42:57.914140 15092 tablet_bootstrap.cc:492] T 11fba61e83294be4b3d4cbe731fe6c4f P d4e3cd899940498d83eba8fc5ed6b9a2: Bootstrap starting.
I20260620 11:42:57.914151 15091 raft_consensus.cc:348] T 11fba61e83294be4b3d4cbe731fe6c4f P 82db96227ab34c6e93967a5951456933 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "580c848b417d4ffbab6d865896c13de7" member_type: VOTER last_known_addr { host: "127.6.234.193" port: 44461 } } peers { permanent_uuid: "82db96227ab34c6e93967a5951456933" member_type: VOTER last_known_addr { host: "127.6.234.194" port: 40195 } } peers { permanent_uuid: "d4e3cd899940498d83eba8fc5ed6b9a2" member_type: VOTER last_known_addr { host: "127.6.234.195" port: 38309 } }
I20260620 11:42:57.914224 15091 raft_consensus.cc:374] T 11fba61e83294be4b3d4cbe731fe6c4f P 82db96227ab34c6e93967a5951456933 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 11:42:57.914249 15091 raft_consensus.cc:729] T 11fba61e83294be4b3d4cbe731fe6c4f P 82db96227ab34c6e93967a5951456933 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 82db96227ab34c6e93967a5951456933, State: Initialized, Role: FOLLOWER
I20260620 11:42:57.914299 15091 consensus_queue.cc:260] T 11fba61e83294be4b3d4cbe731fe6c4f P 82db96227ab34c6e93967a5951456933 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "580c848b417d4ffbab6d865896c13de7" member_type: VOTER last_known_addr { host: "127.6.234.193" port: 44461 } } peers { permanent_uuid: "82db96227ab34c6e93967a5951456933" member_type: VOTER last_known_addr { host: "127.6.234.194" port: 40195 } } peers { permanent_uuid: "d4e3cd899940498d83eba8fc5ed6b9a2" member_type: VOTER last_known_addr { host: "127.6.234.195" port: 38309 } }
I20260620 11:42:57.914392 15091 ts_tablet_manager.cc:1428] T 11fba61e83294be4b3d4cbe731fe6c4f P 82db96227ab34c6e93967a5951456933: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
I20260620 11:42:57.914422 15092 tablet_bootstrap.cc:654] T 11fba61e83294be4b3d4cbe731fe6c4f P d4e3cd899940498d83eba8fc5ed6b9a2: Neither blocks nor log segments found. Creating new log.
I20260620 11:42:57.915119 15092 tablet_bootstrap.cc:492] T 11fba61e83294be4b3d4cbe731fe6c4f P d4e3cd899940498d83eba8fc5ed6b9a2: No bootstrap required, opened a new log
I20260620 11:42:57.915174 15092 ts_tablet_manager.cc:1397] T 11fba61e83294be4b3d4cbe731fe6c4f P d4e3cd899940498d83eba8fc5ed6b9a2: Time spent bootstrapping tablet: real 0.001s user 0.000s sys 0.000s
I20260620 11:42:57.915269 15092 raft_consensus.cc:348] T 11fba61e83294be4b3d4cbe731fe6c4f P d4e3cd899940498d83eba8fc5ed6b9a2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "580c848b417d4ffbab6d865896c13de7" member_type: VOTER last_known_addr { host: "127.6.234.193" port: 44461 } } peers { permanent_uuid: "82db96227ab34c6e93967a5951456933" member_type: VOTER last_known_addr { host: "127.6.234.194" port: 40195 } } peers { permanent_uuid: "d4e3cd899940498d83eba8fc5ed6b9a2" member_type: VOTER last_known_addr { host: "127.6.234.195" port: 38309 } }
I20260620 11:42:57.915340 15092 raft_consensus.cc:374] T 11fba61e83294be4b3d4cbe731fe6c4f P d4e3cd899940498d83eba8fc5ed6b9a2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 11:42:57.915362 15092 raft_consensus.cc:729] T 11fba61e83294be4b3d4cbe731fe6c4f P d4e3cd899940498d83eba8fc5ed6b9a2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d4e3cd899940498d83eba8fc5ed6b9a2, State: Initialized, Role: FOLLOWER
I20260620 11:42:57.915414 15092 consensus_queue.cc:260] T 11fba61e83294be4b3d4cbe731fe6c4f P d4e3cd899940498d83eba8fc5ed6b9a2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "580c848b417d4ffbab6d865896c13de7" member_type: VOTER last_known_addr { host: "127.6.234.193" port: 44461 } } peers { permanent_uuid: "82db96227ab34c6e93967a5951456933" member_type: VOTER last_known_addr { host: "127.6.234.194" port: 40195 } } peers { permanent_uuid: "d4e3cd899940498d83eba8fc5ed6b9a2" member_type: VOTER last_known_addr { host: "127.6.234.195" port: 38309 } }
I20260620 11:42:57.915509 15092 ts_tablet_manager.cc:1428] T 11fba61e83294be4b3d4cbe731fe6c4f P d4e3cd899940498d83eba8fc5ed6b9a2: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
I20260620 11:42:57.997187 15099 raft_consensus.cc:482] T 11fba61e83294be4b3d4cbe731fe6c4f P 580c848b417d4ffbab6d865896c13de7 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260620 11:42:57.997304 15099 raft_consensus.cc:504] T 11fba61e83294be4b3d4cbe731fe6c4f P 580c848b417d4ffbab6d865896c13de7 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "580c848b417d4ffbab6d865896c13de7" member_type: VOTER last_known_addr { host: "127.6.234.193" port: 44461 } } peers { permanent_uuid: "82db96227ab34c6e93967a5951456933" member_type: VOTER last_known_addr { host: "127.6.234.194" port: 40195 } } peers { permanent_uuid: "d4e3cd899940498d83eba8fc5ed6b9a2" member_type: VOTER last_known_addr { host: "127.6.234.195" port: 38309 } }
I20260620 11:42:57.997749 15099 leader_election.cc:288] T 11fba61e83294be4b3d4cbe731fe6c4f P 580c848b417d4ffbab6d865896c13de7 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 82db96227ab34c6e93967a5951456933 (127.6.234.194:40195), d4e3cd899940498d83eba8fc5ed6b9a2 (127.6.234.195:38309)
I20260620 11:42:58.001736 15090 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 11:42:57.998057 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.6.234.193:46915 (local address 127.6.234.195:38309)
0620 11:42:57.998154 (+ 97us) server_negotiation.cc:206] Beginning negotiation
0620 11:42:57.998156 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 11:42:57.998167 (+ 11us) server_negotiation.cc:407] Connection header received
0620 11:42:57.998272 (+ 105us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 11:42:57.998273 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 11:42:57.998283 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 11:42:57.998344 (+ 61us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 11:42:57.998590 (+ 246us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 11:42:57.999158 (+ 568us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 11:42:58.000868 (+ 1710us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 11:42:58.001294 (+ 426us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 11:42:58.001383 (+ 89us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 11:42:58.001394 (+ 11us) server_negotiation.cc:1035] Waiting for connection context
0620 11:42:58.001595 (+ 201us) server_negotiation.cc:299] Negotiation successful
0620 11:42:58.001638 (+ 43us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":76}
I20260620 11:42:58.001734 15088 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 11:42:57.998042 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.6.234.193:54817 (local address 127.6.234.194:40195)
0620 11:42:57.998115 (+ 73us) server_negotiation.cc:206] Beginning negotiation
0620 11:42:57.998117 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 11:42:57.998179 (+ 62us) server_negotiation.cc:407] Connection header received
0620 11:42:57.998272 (+ 93us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 11:42:57.998273 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 11:42:57.998283 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 11:42:57.998346 (+ 63us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 11:42:57.998525 (+ 179us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 11:42:57.999271 (+ 746us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 11:42:58.000737 (+ 1466us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 11:42:58.001305 (+ 568us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 11:42:58.001388 (+ 83us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 11:42:58.001396 (+ 8us) server_negotiation.cc:1035] Waiting for connection context
0620 11:42:58.001610 (+ 214us) server_negotiation.cc:299] Negotiation successful
0620 11:42:58.001646 (+ 36us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":47}
I20260620 11:42:58.001757 15158 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 11:42:57.997934 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.234.194:40195 (local address 127.6.234.193:54817)
0620 11:42:57.998122 (+ 188us) negotiation.cc:107] Waiting for socket to connect
0620 11:42:57.998132 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0620 11:42:57.998204 (+ 72us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 11:42:57.998360 (+ 156us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 11:42:57.998361 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 11:42:57.998364 (+ 3us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 11:42:57.998448 (+ 84us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 11:42:57.998451 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 11:42:57.999368 (+ 917us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 11:42:57.999369 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 11:42:58.000655 (+ 1286us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 11:42:58.000658 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 11:42:58.001445 (+ 787us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 11:42:58.001446 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 11:42:58.001545 (+ 99us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 11:42:58.001549 (+ 4us) client_negotiation.cc:769] Sending connection context
0620 11:42:58.001602 (+ 53us) client_negotiation.cc:240] Negotiation successful
0620 11:42:58.001643 (+ 41us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":177,"thread_start_us":52,"threads_started":1}
I20260620 11:42:58.002202 15159 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 11:42:57.997920 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.234.195:38309 (local address 127.6.234.193:46915)
0620 11:42:57.998103 (+ 183us) negotiation.cc:107] Waiting for socket to connect
0620 11:42:57.998116 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0620 11:42:57.998180 (+ 64us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 11:42:57.998416 (+ 236us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 11:42:57.998417 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 11:42:57.998418 (+ 1us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 11:42:57.998482 (+ 64us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 11:42:57.998483 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 11:42:57.999250 (+ 767us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 11:42:57.999251 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 11:42:58.000777 (+ 1526us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 11:42:58.000779 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 11:42:58.001408 (+ 629us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 11:42:58.001410 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 11:42:58.001486 (+ 76us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 11:42:58.001490 (+ 4us) client_negotiation.cc:769] Sending connection context
0620 11:42:58.002036 (+ 546us) client_negotiation.cc:240] Negotiation successful
0620 11:42:58.002091 (+ 55us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":163,"mutex_wait_us":3,"thread_start_us":68,"threads_started":1}
I20260620 11:42:58.002208 14880 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "11fba61e83294be4b3d4cbe731fe6c4f" candidate_uuid: "580c848b417d4ffbab6d865896c13de7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "82db96227ab34c6e93967a5951456933" is_pre_election: true
I20260620 11:42:58.002298 14880 raft_consensus.cc:2454] T 11fba61e83294be4b3d4cbe731fe6c4f P 82db96227ab34c6e93967a5951456933 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 580c848b417d4ffbab6d865896c13de7 in term 0.
I20260620 11:42:58.002444 15020 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "11fba61e83294be4b3d4cbe731fe6c4f" candidate_uuid: "580c848b417d4ffbab6d865896c13de7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d4e3cd899940498d83eba8fc5ed6b9a2" is_pre_election: true
I20260620 11:42:58.002475 14674 leader_election.cc:302] T 11fba61e83294be4b3d4cbe731fe6c4f P 580c848b417d4ffbab6d865896c13de7 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 580c848b417d4ffbab6d865896c13de7, 82db96227ab34c6e93967a5951456933; no voters:
I20260620 11:42:58.002537 15020 raft_consensus.cc:2454] T 11fba61e83294be4b3d4cbe731fe6c4f P d4e3cd899940498d83eba8fc5ed6b9a2 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 580c848b417d4ffbab6d865896c13de7 in term 0.
I20260620 11:42:58.002590 15099 raft_consensus.cc:2781] T 11fba61e83294be4b3d4cbe731fe6c4f P 580c848b417d4ffbab6d865896c13de7 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260620 11:42:58.002631 15099 raft_consensus.cc:482] T 11fba61e83294be4b3d4cbe731fe6c4f P 580c848b417d4ffbab6d865896c13de7 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260620 11:42:58.002656 15099 raft_consensus.cc:3037] T 11fba61e83294be4b3d4cbe731fe6c4f P 580c848b417d4ffbab6d865896c13de7 [term 0 FOLLOWER]: Advancing to term 1
I20260620 11:42:58.003297 15099 raft_consensus.cc:504] T 11fba61e83294be4b3d4cbe731fe6c4f P 580c848b417d4ffbab6d865896c13de7 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "580c848b417d4ffbab6d865896c13de7" member_type: VOTER last_known_addr { host: "127.6.234.193" port: 44461 } } peers { permanent_uuid: "82db96227ab34c6e93967a5951456933" member_type: VOTER last_known_addr { host: "127.6.234.194" port: 40195 } } peers { permanent_uuid: "d4e3cd899940498d83eba8fc5ed6b9a2" member_type: VOTER last_known_addr { host: "127.6.234.195" port: 38309 } }
I20260620 11:42:58.003439 15099 leader_election.cc:288] T 11fba61e83294be4b3d4cbe731fe6c4f P 580c848b417d4ffbab6d865896c13de7 [CANDIDATE]: Term 1 election: Requested vote from peers 82db96227ab34c6e93967a5951456933 (127.6.234.194:40195), d4e3cd899940498d83eba8fc5ed6b9a2 (127.6.234.195:38309)
I20260620 11:42:58.003573 14880 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "11fba61e83294be4b3d4cbe731fe6c4f" candidate_uuid: "580c848b417d4ffbab6d865896c13de7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "82db96227ab34c6e93967a5951456933"
I20260620 11:42:58.003648 14880 raft_consensus.cc:3037] T 11fba61e83294be4b3d4cbe731fe6c4f P 82db96227ab34c6e93967a5951456933 [term 0 FOLLOWER]: Advancing to term 1
I20260620 11:42:58.003624 15020 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "11fba61e83294be4b3d4cbe731fe6c4f" candidate_uuid: "580c848b417d4ffbab6d865896c13de7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d4e3cd899940498d83eba8fc5ed6b9a2"
I20260620 11:42:58.003695 15020 raft_consensus.cc:3037] T 11fba61e83294be4b3d4cbe731fe6c4f P d4e3cd899940498d83eba8fc5ed6b9a2 [term 0 FOLLOWER]: Advancing to term 1
I20260620 11:42:58.004220 14880 raft_consensus.cc:2454] T 11fba61e83294be4b3d4cbe731fe6c4f P 82db96227ab34c6e93967a5951456933 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 580c848b417d4ffbab6d865896c13de7 in term 1.
I20260620 11:42:58.004491 15020 raft_consensus.cc:2454] T 11fba61e83294be4b3d4cbe731fe6c4f P d4e3cd899940498d83eba8fc5ed6b9a2 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 580c848b417d4ffbab6d865896c13de7 in term 1.
I20260620 11:42:58.004669 14673 leader_election.cc:302] T 11fba61e83294be4b3d4cbe731fe6c4f P 580c848b417d4ffbab6d865896c13de7 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 580c848b417d4ffbab6d865896c13de7, d4e3cd899940498d83eba8fc5ed6b9a2; no voters:
I20260620 11:42:58.004786 15099 raft_consensus.cc:2781] T 11fba61e83294be4b3d4cbe731fe6c4f P 580c848b417d4ffbab6d865896c13de7 [term 1 FOLLOWER]: Leader election won for term 1
I20260620 11:42:58.005008 15099 raft_consensus.cc:686] T 11fba61e83294be4b3d4cbe731fe6c4f P 580c848b417d4ffbab6d865896c13de7 [term 1 LEADER]: Becoming Leader. State: Replica: 580c848b417d4ffbab6d865896c13de7, State: Running, Role: LEADER
I20260620 11:42:58.005091 15099 consensus_queue.cc:237] T 11fba61e83294be4b3d4cbe731fe6c4f P 580c848b417d4ffbab6d865896c13de7 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "580c848b417d4ffbab6d865896c13de7" member_type: VOTER last_known_addr { host: "127.6.234.193" port: 44461 } } peers { permanent_uuid: "82db96227ab34c6e93967a5951456933" member_type: VOTER last_known_addr { host: "127.6.234.194" port: 40195 } } peers { permanent_uuid: "d4e3cd899940498d83eba8fc5ed6b9a2" member_type: VOTER last_known_addr { host: "127.6.234.195" port: 38309 } }
I20260620 11:42:58.005828 14583 catalog_manager.cc:5495] T 11fba61e83294be4b3d4cbe731fe6c4f P 580c848b417d4ffbab6d865896c13de7 reported cstate change: term changed from 0 to 1, leader changed from <none> to 580c848b417d4ffbab6d865896c13de7 (127.6.234.193). New cstate: current_term: 1 leader_uuid: "580c848b417d4ffbab6d865896c13de7" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "580c848b417d4ffbab6d865896c13de7" member_type: VOTER last_known_addr { host: "127.6.234.193" port: 44461 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "82db96227ab34c6e93967a5951456933" member_type: VOTER last_known_addr { host: "127.6.234.194" port: 40195 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "d4e3cd899940498d83eba8fc5ed6b9a2" member_type: VOTER last_known_addr { host: "127.6.234.195" port: 38309 } health_report { overall_health: UNKNOWN } } }
I20260620 11:42:58.032970 15089 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 11:42:58.030419 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40872 (local address 127.6.234.193:44461)
0620 11:42:58.030477 (+ 58us) server_negotiation.cc:206] Beginning negotiation
0620 11:42:58.030479 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 11:42:58.030489 (+ 10us) server_negotiation.cc:407] Connection header received
0620 11:42:58.030597 (+ 108us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 11:42:58.030598 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 11:42:58.030620 (+ 22us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 11:42:58.030725 (+ 105us) server_negotiation.cc:226] Negotiated authn=TOKEN
0620 11:42:58.030910 (+ 185us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 11:42:58.031395 (+ 485us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 11:42:58.032149 (+ 754us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 11:42:58.032368 (+ 219us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 11:42:58.032446 (+ 78us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 11:42:58.032608 (+ 162us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0620 11:42:58.032726 (+ 118us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0620 11:42:58.032813 (+ 87us) server_negotiation.cc:1035] Waiting for connection context
0620 11:42:58.032867 (+ 54us) server_negotiation.cc:299] Negotiation successful
0620 11:42:58.032885 (+ 18us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":36}
I20260620 11:42:58.033953 15020 raft_consensus.cc:1261] T 11fba61e83294be4b3d4cbe731fe6c4f P d4e3cd899940498d83eba8fc5ed6b9a2 [term 1 FOLLOWER]: Refusing update from remote peer 580c848b417d4ffbab6d865896c13de7: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 11:42:58.033965 14880 raft_consensus.cc:1261] T 11fba61e83294be4b3d4cbe731fe6c4f P 82db96227ab34c6e93967a5951456933 [term 1 FOLLOWER]: Refusing update from remote peer 580c848b417d4ffbab6d865896c13de7: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 11:42:58.034226 15160 consensus_queue.cc:1035] T 11fba61e83294be4b3d4cbe731fe6c4f P 580c848b417d4ffbab6d865896c13de7 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d4e3cd899940498d83eba8fc5ed6b9a2" member_type: VOTER last_known_addr { host: "127.6.234.195" port: 38309 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 11:42:58.034374 15099 consensus_queue.cc:1035] T 11fba61e83294be4b3d4cbe731fe6c4f P 580c848b417d4ffbab6d865896c13de7 [LEADER]: Connected to new peer: Peer: permanent_uuid: "82db96227ab34c6e93967a5951456933" member_type: VOTER last_known_addr { host: "127.6.234.194" port: 40195 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 11:42:58.035560 15106 mvcc.cc:205] Tried to move back new op lower bound from 7298890866825076736 to 7298890866709389312. Current Snapshot: MvccSnapshot[applied={T|T < 7298890866825076736}]
I20260620 11:42:58.037843 14583 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:46562:
table { table_name: "test-table" } modify_external_catalogs: true
I20260620 11:42:58.037926 14583 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:46562:
table { table_name: "test-table" } modify_external_catalogs: true
I20260620 11:42:58.038552 14583 catalog_manager.cc:5782] T 00000000000000000000000000000000 P f75e5ca955994ed9ba7f8b6ce11b9c5a: Sending DeleteTablet for 3 replicas of tablet 11fba61e83294be4b3d4cbe731fe6c4f
I20260620 11:42:58.038793 14859 tablet_service.cc:1514] Processing DeleteTablet for tablet 11fba61e83294be4b3d4cbe731fe6c4f with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 11:42:58 UTC) from {username='oryx', principal='oryx/127.6.234.254@KRBTEST.COM'} at 127.0.0.1:39656
I20260620 11:42:58.038894 15113 tablet_replica.cc:330] T 11fba61e83294be4b3d4cbe731fe6c4f P 82db96227ab34c6e93967a5951456933: stopping tablet replica
I20260620 11:42:58.038949 15113 raft_consensus.cc:2229] T 11fba61e83294be4b3d4cbe731fe6c4f P 82db96227ab34c6e93967a5951456933 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260620 11:42:58.038991 15113 raft_consensus.cc:2258] T 11fba61e83294be4b3d4cbe731fe6c4f P 82db96227ab34c6e93967a5951456933 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 11:42:58.038991 14720 tablet_service.cc:1514] Processing DeleteTablet for tablet 11fba61e83294be4b3d4cbe731fe6c4f with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 11:42:58 UTC) from {username='oryx', principal='oryx/127.6.234.254@KRBTEST.COM'} at 127.0.0.1:40862
I20260620 11:42:58.039129 15115 tablet_replica.cc:330] T 11fba61e83294be4b3d4cbe731fe6c4f P 580c848b417d4ffbab6d865896c13de7: stopping tablet replica
I20260620 11:42:58.039186 15115 raft_consensus.cc:2229] T 11fba61e83294be4b3d4cbe731fe6c4f P 580c848b417d4ffbab6d865896c13de7 [term 1 LEADER]: Raft consensus shutting down.
I20260620 11:42:58.039273 15115 raft_consensus.cc:2258] T 11fba61e83294be4b3d4cbe731fe6c4f P 580c848b417d4ffbab6d865896c13de7 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 11:42:58.039690 15113 ts_tablet_manager.cc:1905] T 11fba61e83294be4b3d4cbe731fe6c4f P 82db96227ab34c6e93967a5951456933: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 11:42:58.041309 15115 ts_tablet_manager.cc:1905] T 11fba61e83294be4b3d4cbe731fe6c4f P 580c848b417d4ffbab6d865896c13de7: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 11:42:58.041517 15113 ts_tablet_manager.cc:1918] T 11fba61e83294be4b3d4cbe731fe6c4f P 82db96227ab34c6e93967a5951456933: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260620 11:42:58.041575 15113 log.cc:1196] T 11fba61e83294be4b3d4cbe731fe6c4f P 82db96227ab34c6e93967a5951456933: Deleting WAL directory at /tmp/dist-test-task1JRg7N/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781955746769051-7083-0/minicluster-data/ts-1/wal/wals/11fba61e83294be4b3d4cbe731fe6c4f
I20260620 11:42:58.041827 15113 ts_tablet_manager.cc:1939] T 11fba61e83294be4b3d4cbe731fe6c4f P 82db96227ab34c6e93967a5951456933: Deleting consensus metadata
I20260620 11:42:58.042759 15115 ts_tablet_manager.cc:1918] T 11fba61e83294be4b3d4cbe731fe6c4f P 580c848b417d4ffbab6d865896c13de7: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260620 11:42:58.042821 15115 log.cc:1196] T 11fba61e83294be4b3d4cbe731fe6c4f P 580c848b417d4ffbab6d865896c13de7: Deleting WAL directory at /tmp/dist-test-task1JRg7N/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781955746769051-7083-0/minicluster-data/ts-0/wal/wals/11fba61e83294be4b3d4cbe731fe6c4f
I20260620 11:42:58.043038 15115 ts_tablet_manager.cc:1939] T 11fba61e83294be4b3d4cbe731fe6c4f P 580c848b417d4ffbab6d865896c13de7: Deleting consensus metadata
I20260620 11:42:58.043262 14570 catalog_manager.cc:4841] TS 82db96227ab34c6e93967a5951456933 (127.6.234.194:40195): tablet 11fba61e83294be4b3d4cbe731fe6c4f (table test-table [id=3fb436b92b044e269ad11ad1598cee8b]) successfully deleted
I20260620 11:42:58.043396 14567 catalog_manager.cc:4841] TS 580c848b417d4ffbab6d865896c13de7 (127.6.234.193:44461): tablet 11fba61e83294be4b3d4cbe731fe6c4f (table test-table [id=3fb436b92b044e269ad11ad1598cee8b]) successfully deleted
I20260620 11:42:58.043529 15000 tablet_service.cc:1514] Processing DeleteTablet for tablet 11fba61e83294be4b3d4cbe731fe6c4f with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 11:42:58 UTC) from {username='oryx', principal='oryx/127.6.234.254@KRBTEST.COM'} at 127.0.0.1:44454
I20260620 11:42:58.043617 15114 tablet_replica.cc:330] T 11fba61e83294be4b3d4cbe731fe6c4f P d4e3cd899940498d83eba8fc5ed6b9a2: stopping tablet replica
I20260620 11:42:58.043669 15114 raft_consensus.cc:2229] T 11fba61e83294be4b3d4cbe731fe6c4f P d4e3cd899940498d83eba8fc5ed6b9a2 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260620 11:42:58.043709 15114 raft_consensus.cc:2258] T 11fba61e83294be4b3d4cbe731fe6c4f P d4e3cd899940498d83eba8fc5ed6b9a2 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 11:42:58.044004 14583 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:46562:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260620 11:42:58.044139 14583 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260620 11:42:58.045864 14720 tablet_service.cc:1467] Processing CreateTablet for tablet 04ff4b4ec62e40d0a9cd57d29da28d20 (DEFAULT_TABLE table=test-table [id=b0538bdf02bc41cda8d0a3a823c8e0ee]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 11:42:58.046017 14720 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 04ff4b4ec62e40d0a9cd57d29da28d20. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 11:42:58.047237 15093 tablet_bootstrap.cc:492] T 04ff4b4ec62e40d0a9cd57d29da28d20 P 580c848b417d4ffbab6d865896c13de7: Bootstrap starting.
I20260620 11:42:58.047469 14859 tablet_service.cc:1467] Processing CreateTablet for tablet 04ff4b4ec62e40d0a9cd57d29da28d20 (DEFAULT_TABLE table=test-table [id=b0538bdf02bc41cda8d0a3a823c8e0ee]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 11:42:58.047571 14859 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 04ff4b4ec62e40d0a9cd57d29da28d20. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 11:42:58.047633 15093 tablet_bootstrap.cc:654] T 04ff4b4ec62e40d0a9cd57d29da28d20 P 580c848b417d4ffbab6d865896c13de7: Neither blocks nor log segments found. Creating new log.
I20260620 11:42:58.048835 15091 tablet_bootstrap.cc:492] T 04ff4b4ec62e40d0a9cd57d29da28d20 P 82db96227ab34c6e93967a5951456933: Bootstrap starting.
I20260620 11:42:58.049135 15091 tablet_bootstrap.cc:654] T 04ff4b4ec62e40d0a9cd57d29da28d20 P 82db96227ab34c6e93967a5951456933: Neither blocks nor log segments found. Creating new log.
I20260620 11:42:58.049113 15000 tablet_service.cc:1467] Processing CreateTablet for tablet 04ff4b4ec62e40d0a9cd57d29da28d20 (DEFAULT_TABLE table=test-table [id=b0538bdf02bc41cda8d0a3a823c8e0ee]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 11:42:58.049221 15000 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 04ff4b4ec62e40d0a9cd57d29da28d20. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 11:42:58.049743 15091 tablet_bootstrap.cc:492] T 04ff4b4ec62e40d0a9cd57d29da28d20 P 82db96227ab34c6e93967a5951456933: No bootstrap required, opened a new log
I20260620 11:42:58.049790 15091 ts_tablet_manager.cc:1397] T 04ff4b4ec62e40d0a9cd57d29da28d20 P 82db96227ab34c6e93967a5951456933: Time spent bootstrapping tablet: real 0.001s user 0.001s sys 0.000s
I20260620 11:42:58.049908 15091 raft_consensus.cc:348] T 04ff4b4ec62e40d0a9cd57d29da28d20 P 82db96227ab34c6e93967a5951456933 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "580c848b417d4ffbab6d865896c13de7" member_type: VOTER last_known_addr { host: "127.6.234.193" port: 44461 } } peers { permanent_uuid: "82db96227ab34c6e93967a5951456933" member_type: VOTER last_known_addr { host: "127.6.234.194" port: 40195 } } peers { permanent_uuid: "d4e3cd899940498d83eba8fc5ed6b9a2" member_type: VOTER last_known_addr { host: "127.6.234.195" port: 38309 } }
I20260620 11:42:58.049966 15091 raft_consensus.cc:374] T 04ff4b4ec62e40d0a9cd57d29da28d20 P 82db96227ab34c6e93967a5951456933 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 11:42:58.049986 15091 raft_consensus.cc:729] T 04ff4b4ec62e40d0a9cd57d29da28d20 P 82db96227ab34c6e93967a5951456933 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 82db96227ab34c6e93967a5951456933, State: Initialized, Role: FOLLOWER
I20260620 11:42:58.050034 15091 consensus_queue.cc:260] T 04ff4b4ec62e40d0a9cd57d29da28d20 P 82db96227ab34c6e93967a5951456933 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "580c848b417d4ffbab6d865896c13de7" member_type: VOTER last_known_addr { host: "127.6.234.193" port: 44461 } } peers { permanent_uuid: "82db96227ab34c6e93967a5951456933" member_type: VOTER last_known_addr { host: "127.6.234.194" port: 40195 } } peers { permanent_uuid: "d4e3cd899940498d83eba8fc5ed6b9a2" member_type: VOTER last_known_addr { host: "127.6.234.195" port: 38309 } }
I20260620 11:42:58.050132 15091 ts_tablet_manager.cc:1428] T 04ff4b4ec62e40d0a9cd57d29da28d20 P 82db96227ab34c6e93967a5951456933: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
I20260620 11:42:58.050549 15092 tablet_bootstrap.cc:492] T 04ff4b4ec62e40d0a9cd57d29da28d20 P d4e3cd899940498d83eba8fc5ed6b9a2: Bootstrap starting.
I20260620 11:42:58.050913 15092 tablet_bootstrap.cc:654] T 04ff4b4ec62e40d0a9cd57d29da28d20 P d4e3cd899940498d83eba8fc5ed6b9a2: Neither blocks nor log segments found. Creating new log.
I20260620 11:42:58.051607 15114 ts_tablet_manager.cc:1905] T 11fba61e83294be4b3d4cbe731fe6c4f P d4e3cd899940498d83eba8fc5ed6b9a2: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 11:42:58.052047 15092 tablet_bootstrap.cc:492] T 04ff4b4ec62e40d0a9cd57d29da28d20 P d4e3cd899940498d83eba8fc5ed6b9a2: No bootstrap required, opened a new log
I20260620 11:42:58.052089 15092 ts_tablet_manager.cc:1397] T 04ff4b4ec62e40d0a9cd57d29da28d20 P d4e3cd899940498d83eba8fc5ed6b9a2: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260620 11:42:58.052208 15092 raft_consensus.cc:348] T 04ff4b4ec62e40d0a9cd57d29da28d20 P d4e3cd899940498d83eba8fc5ed6b9a2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "580c848b417d4ffbab6d865896c13de7" member_type: VOTER last_known_addr { host: "127.6.234.193" port: 44461 } } peers { permanent_uuid: "82db96227ab34c6e93967a5951456933" member_type: VOTER last_known_addr { host: "127.6.234.194" port: 40195 } } peers { permanent_uuid: "d4e3cd899940498d83eba8fc5ed6b9a2" member_type: VOTER last_known_addr { host: "127.6.234.195" port: 38309 } }
I20260620 11:42:58.052275 15092 raft_consensus.cc:374] T 04ff4b4ec62e40d0a9cd57d29da28d20 P d4e3cd899940498d83eba8fc5ed6b9a2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 11:42:58.052295 15092 raft_consensus.cc:729] T 04ff4b4ec62e40d0a9cd57d29da28d20 P d4e3cd899940498d83eba8fc5ed6b9a2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d4e3cd899940498d83eba8fc5ed6b9a2, State: Initialized, Role: FOLLOWER
I20260620 11:42:58.052337 15092 consensus_queue.cc:260] T 04ff4b4ec62e40d0a9cd57d29da28d20 P d4e3cd899940498d83eba8fc5ed6b9a2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "580c848b417d4ffbab6d865896c13de7" member_type: VOTER last_known_addr { host: "127.6.234.193" port: 44461 } } peers { permanent_uuid: "82db96227ab34c6e93967a5951456933" member_type: VOTER last_known_addr { host: "127.6.234.194" port: 40195 } } peers { permanent_uuid: "d4e3cd899940498d83eba8fc5ed6b9a2" member_type: VOTER last_known_addr { host: "127.6.234.195" port: 38309 } }
I20260620 11:42:58.052421 15092 ts_tablet_manager.cc:1428] T 04ff4b4ec62e40d0a9cd57d29da28d20 P d4e3cd899940498d83eba8fc5ed6b9a2: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
I20260620 11:42:58.052922 15114 ts_tablet_manager.cc:1918] T 11fba61e83294be4b3d4cbe731fe6c4f P d4e3cd899940498d83eba8fc5ed6b9a2: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260620 11:42:58.052970 15114 log.cc:1196] T 11fba61e83294be4b3d4cbe731fe6c4f P d4e3cd899940498d83eba8fc5ed6b9a2: Deleting WAL directory at /tmp/dist-test-task1JRg7N/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781955746769051-7083-0/minicluster-data/ts-2/wal/wals/11fba61e83294be4b3d4cbe731fe6c4f
I20260620 11:42:58.053176 15114 ts_tablet_manager.cc:1939] T 11fba61e83294be4b3d4cbe731fe6c4f P d4e3cd899940498d83eba8fc5ed6b9a2: Deleting consensus metadata
I20260620 11:42:58.053460 15093 tablet_bootstrap.cc:492] T 04ff4b4ec62e40d0a9cd57d29da28d20 P 580c848b417d4ffbab6d865896c13de7: No bootstrap required, opened a new log
I20260620 11:42:58.053501 15093 ts_tablet_manager.cc:1397] T 04ff4b4ec62e40d0a9cd57d29da28d20 P 580c848b417d4ffbab6d865896c13de7: Time spent bootstrapping tablet: real 0.006s user 0.000s sys 0.001s
I20260620 11:42:58.053578 14569 catalog_manager.cc:4841] TS d4e3cd899940498d83eba8fc5ed6b9a2 (127.6.234.195:38309): tablet 11fba61e83294be4b3d4cbe731fe6c4f (table test-table [id=3fb436b92b044e269ad11ad1598cee8b]) successfully deleted
I20260620 11:42:58.053632 15093 raft_consensus.cc:348] T 04ff4b4ec62e40d0a9cd57d29da28d20 P 580c848b417d4ffbab6d865896c13de7 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "580c848b417d4ffbab6d865896c13de7" member_type: VOTER last_known_addr { host: "127.6.234.193" port: 44461 } } peers { permanent_uuid: "82db96227ab34c6e93967a5951456933" member_type: VOTER last_known_addr { host: "127.6.234.194" port: 40195 } } peers { permanent_uuid: "d4e3cd899940498d83eba8fc5ed6b9a2" member_type: VOTER last_known_addr { host: "127.6.234.195" port: 38309 } }
I20260620 11:42:58.053730 15093 raft_consensus.cc:374] T 04ff4b4ec62e40d0a9cd57d29da28d20 P 580c848b417d4ffbab6d865896c13de7 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 11:42:58.053752 15093 raft_consensus.cc:729] T 04ff4b4ec62e40d0a9cd57d29da28d20 P 580c848b417d4ffbab6d865896c13de7 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 580c848b417d4ffbab6d865896c13de7, State: Initialized, Role: FOLLOWER
I20260620 11:42:58.053794 15093 consensus_queue.cc:260] T 04ff4b4ec62e40d0a9cd57d29da28d20 P 580c848b417d4ffbab6d865896c13de7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "580c848b417d4ffbab6d865896c13de7" member_type: VOTER last_known_addr { host: "127.6.234.193" port: 44461 } } peers { permanent_uuid: "82db96227ab34c6e93967a5951456933" member_type: VOTER last_known_addr { host: "127.6.234.194" port: 40195 } } peers { permanent_uuid: "d4e3cd899940498d83eba8fc5ed6b9a2" member_type: VOTER last_known_addr { host: "127.6.234.195" port: 38309 } }
I20260620 11:42:58.053885 15093 ts_tablet_manager.cc:1428] T 04ff4b4ec62e40d0a9cd57d29da28d20 P 580c848b417d4ffbab6d865896c13de7: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
I20260620 11:42:58.130364 15160 raft_consensus.cc:482] T 04ff4b4ec62e40d0a9cd57d29da28d20 P 580c848b417d4ffbab6d865896c13de7 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260620 11:42:58.130478 15160 raft_consensus.cc:504] T 04ff4b4ec62e40d0a9cd57d29da28d20 P 580c848b417d4ffbab6d865896c13de7 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "580c848b417d4ffbab6d865896c13de7" member_type: VOTER last_known_addr { host: "127.6.234.193" port: 44461 } } peers { permanent_uuid: "82db96227ab34c6e93967a5951456933" member_type: VOTER last_known_addr { host: "127.6.234.194" port: 40195 } } peers { permanent_uuid: "d4e3cd899940498d83eba8fc5ed6b9a2" member_type: VOTER last_known_addr { host: "127.6.234.195" port: 38309 } }
I20260620 11:42:58.130681 15160 leader_election.cc:288] T 04ff4b4ec62e40d0a9cd57d29da28d20 P 580c848b417d4ffbab6d865896c13de7 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 82db96227ab34c6e93967a5951456933 (127.6.234.194:40195), d4e3cd899940498d83eba8fc5ed6b9a2 (127.6.234.195:38309)
I20260620 11:42:58.130873 15020 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "04ff4b4ec62e40d0a9cd57d29da28d20" candidate_uuid: "580c848b417d4ffbab6d865896c13de7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d4e3cd899940498d83eba8fc5ed6b9a2" is_pre_election: true
I20260620 11:42:58.130885 14880 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "04ff4b4ec62e40d0a9cd57d29da28d20" candidate_uuid: "580c848b417d4ffbab6d865896c13de7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "82db96227ab34c6e93967a5951456933" is_pre_election: true
I20260620 11:42:58.130980 14880 raft_consensus.cc:2454] T 04ff4b4ec62e40d0a9cd57d29da28d20 P 82db96227ab34c6e93967a5951456933 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 580c848b417d4ffbab6d865896c13de7 in term 0.
I20260620 11:42:58.130987 15020 raft_consensus.cc:2454] T 04ff4b4ec62e40d0a9cd57d29da28d20 P d4e3cd899940498d83eba8fc5ed6b9a2 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 580c848b417d4ffbab6d865896c13de7 in term 0.
I20260620 11:42:58.131130 14674 leader_election.cc:302] T 04ff4b4ec62e40d0a9cd57d29da28d20 P 580c848b417d4ffbab6d865896c13de7 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 580c848b417d4ffbab6d865896c13de7, 82db96227ab34c6e93967a5951456933; no voters:
I20260620 11:42:58.131245 15160 raft_consensus.cc:2781] T 04ff4b4ec62e40d0a9cd57d29da28d20 P 580c848b417d4ffbab6d865896c13de7 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260620 11:42:58.131309 15160 raft_consensus.cc:482] T 04ff4b4ec62e40d0a9cd57d29da28d20 P 580c848b417d4ffbab6d865896c13de7 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260620 11:42:58.131330 15160 raft_consensus.cc:3037] T 04ff4b4ec62e40d0a9cd57d29da28d20 P 580c848b417d4ffbab6d865896c13de7 [term 0 FOLLOWER]: Advancing to term 1
I20260620 11:42:58.132287 15160 raft_consensus.cc:504] T 04ff4b4ec62e40d0a9cd57d29da28d20 P 580c848b417d4ffbab6d865896c13de7 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "580c848b417d4ffbab6d865896c13de7" member_type: VOTER last_known_addr { host: "127.6.234.193" port: 44461 } } peers { permanent_uuid: "82db96227ab34c6e93967a5951456933" member_type: VOTER last_known_addr { host: "127.6.234.194" port: 40195 } } peers { permanent_uuid: "d4e3cd899940498d83eba8fc5ed6b9a2" member_type: VOTER last_known_addr { host: "127.6.234.195" port: 38309 } }
I20260620 11:42:58.132423 15160 leader_election.cc:288] T 04ff4b4ec62e40d0a9cd57d29da28d20 P 580c848b417d4ffbab6d865896c13de7 [CANDIDATE]: Term 1 election: Requested vote from peers 82db96227ab34c6e93967a5951456933 (127.6.234.194:40195), d4e3cd899940498d83eba8fc5ed6b9a2 (127.6.234.195:38309)
I20260620 11:42:58.132589 15020 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "04ff4b4ec62e40d0a9cd57d29da28d20" candidate_uuid: "580c848b417d4ffbab6d865896c13de7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d4e3cd899940498d83eba8fc5ed6b9a2"
I20260620 11:42:58.132613 14880 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "04ff4b4ec62e40d0a9cd57d29da28d20" candidate_uuid: "580c848b417d4ffbab6d865896c13de7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "82db96227ab34c6e93967a5951456933"
I20260620 11:42:58.132695 14880 raft_consensus.cc:3037] T 04ff4b4ec62e40d0a9cd57d29da28d20 P 82db96227ab34c6e93967a5951456933 [term 0 FOLLOWER]: Advancing to term 1
I20260620 11:42:58.132694 15020 raft_consensus.cc:3037] T 04ff4b4ec62e40d0a9cd57d29da28d20 P d4e3cd899940498d83eba8fc5ed6b9a2 [term 0 FOLLOWER]: Advancing to term 1
I20260620 11:42:58.133322 15020 raft_consensus.cc:2454] T 04ff4b4ec62e40d0a9cd57d29da28d20 P d4e3cd899940498d83eba8fc5ed6b9a2 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 580c848b417d4ffbab6d865896c13de7 in term 1.
I20260620 11:42:58.133487 14673 leader_election.cc:302] T 04ff4b4ec62e40d0a9cd57d29da28d20 P 580c848b417d4ffbab6d865896c13de7 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 580c848b417d4ffbab6d865896c13de7, d4e3cd899940498d83eba8fc5ed6b9a2; no voters:
I20260620 11:42:58.133579 14880 raft_consensus.cc:2454] T 04ff4b4ec62e40d0a9cd57d29da28d20 P 82db96227ab34c6e93967a5951456933 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 580c848b417d4ffbab6d865896c13de7 in term 1.
I20260620 11:42:58.133772 15160 raft_consensus.cc:2781] T 04ff4b4ec62e40d0a9cd57d29da28d20 P 580c848b417d4ffbab6d865896c13de7 [term 1 FOLLOWER]: Leader election won for term 1
I20260620 11:42:58.133826 15160 raft_consensus.cc:686] T 04ff4b4ec62e40d0a9cd57d29da28d20 P 580c848b417d4ffbab6d865896c13de7 [term 1 LEADER]: Becoming Leader. State: Replica: 580c848b417d4ffbab6d865896c13de7, State: Running, Role: LEADER
I20260620 11:42:58.133878 15160 consensus_queue.cc:237] T 04ff4b4ec62e40d0a9cd57d29da28d20 P 580c848b417d4ffbab6d865896c13de7 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "580c848b417d4ffbab6d865896c13de7" member_type: VOTER last_known_addr { host: "127.6.234.193" port: 44461 } } peers { permanent_uuid: "82db96227ab34c6e93967a5951456933" member_type: VOTER last_known_addr { host: "127.6.234.194" port: 40195 } } peers { permanent_uuid: "d4e3cd899940498d83eba8fc5ed6b9a2" member_type: VOTER last_known_addr { host: "127.6.234.195" port: 38309 } }
I20260620 11:42:58.134500 14583 catalog_manager.cc:5495] T 04ff4b4ec62e40d0a9cd57d29da28d20 P 580c848b417d4ffbab6d865896c13de7 reported cstate change: term changed from 0 to 1, leader changed from <none> to 580c848b417d4ffbab6d865896c13de7 (127.6.234.193). New cstate: current_term: 1 leader_uuid: "580c848b417d4ffbab6d865896c13de7" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "580c848b417d4ffbab6d865896c13de7" member_type: VOTER last_known_addr { host: "127.6.234.193" port: 44461 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "82db96227ab34c6e93967a5951456933" member_type: VOTER last_known_addr { host: "127.6.234.194" port: 40195 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "d4e3cd899940498d83eba8fc5ed6b9a2" member_type: VOTER last_known_addr { host: "127.6.234.195" port: 38309 } health_report { overall_health: UNKNOWN } } }
I20260620 11:42:58.152064 15173 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 11:42:58.149084 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.234.194:40195 (local address 127.6.234.193:34811)
0620 11:42:58.149205 (+ 121us) negotiation.cc:107] Waiting for socket to connect
0620 11:42:58.149215 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0620 11:42:58.149273 (+ 58us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 11:42:58.149396 (+ 123us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 11:42:58.149396 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 11:42:58.149397 (+ 1us) client_negotiation.cc:189] Negotiated authn=TOKEN
0620 11:42:58.149497 (+ 100us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 11:42:58.149499 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 11:42:58.150530 (+ 1031us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 11:42:58.150531 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 11:42:58.151170 (+ 639us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 11:42:58.151172 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 11:42:58.151680 (+ 508us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 11:42:58.151681 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 11:42:58.151734 (+ 53us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 11:42:58.151746 (+ 12us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0620 11:42:58.151923 (+ 177us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0620 11:42:58.151925 (+ 2us) client_negotiation.cc:769] Sending connection context
0620 11:42:58.151959 (+ 34us) client_negotiation.cc:240] Negotiation successful
0620 11:42:58.151995 (+ 36us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":86,"thread_start_us":61,"threads_started":1}
I20260620 11:42:58.152060 15088 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 11:42:58.149117 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.6.234.193:34811 (local address 127.6.234.194:40195)
0620 11:42:58.149168 (+ 51us) server_negotiation.cc:206] Beginning negotiation
0620 11:42:58.149171 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0620 11:42:58.149284 (+ 113us) server_negotiation.cc:407] Connection header received
0620 11:42:58.149313 (+ 29us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 11:42:58.149313 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 11:42:58.149324 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 11:42:58.149416 (+ 92us) server_negotiation.cc:226] Negotiated authn=TOKEN
0620 11:42:58.149580 (+ 164us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 11:42:58.150443 (+ 863us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 11:42:58.151250 (+ 807us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 11:42:58.151582 (+ 332us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 11:42:58.151683 (+ 101us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 11:42:58.151830 (+ 147us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0620 11:42:58.151865 (+ 35us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0620 11:42:58.151918 (+ 53us) server_negotiation.cc:1035] Waiting for connection context
0620 11:42:58.151962 (+ 44us) server_negotiation.cc:299] Negotiation successful
0620 11:42:58.151983 (+ 21us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":27}
I20260620 11:42:58.154109 15020 raft_consensus.cc:1261] T 04ff4b4ec62e40d0a9cd57d29da28d20 P d4e3cd899940498d83eba8fc5ed6b9a2 [term 1 FOLLOWER]: Refusing update from remote peer 580c848b417d4ffbab6d865896c13de7: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 11:42:58.154138 14880 raft_consensus.cc:1261] T 04ff4b4ec62e40d0a9cd57d29da28d20 P 82db96227ab34c6e93967a5951456933 [term 1 FOLLOWER]: Refusing update from remote peer 580c848b417d4ffbab6d865896c13de7: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 11:42:58.154347 15165 consensus_queue.cc:1035] T 04ff4b4ec62e40d0a9cd57d29da28d20 P 580c848b417d4ffbab6d865896c13de7 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d4e3cd899940498d83eba8fc5ed6b9a2" member_type: VOTER last_known_addr { host: "127.6.234.195" port: 38309 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 11:42:58.154461 15160 consensus_queue.cc:1035] T 04ff4b4ec62e40d0a9cd57d29da28d20 P 580c848b417d4ffbab6d865896c13de7 [LEADER]: Connected to new peer: Peer: permanent_uuid: "82db96227ab34c6e93967a5951456933" member_type: VOTER last_known_addr { host: "127.6.234.194" port: 40195 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 11:42:58.161329 14933 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 11:42:58.158924 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.234.193:44461 (local address 127.6.234.194:35621)
0620 11:42:58.158964 (+ 40us) negotiation.cc:107] Waiting for socket to connect
0620 11:42:58.158977 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0620 11:42:58.159021 (+ 44us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 11:42:58.159209 (+ 188us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 11:42:58.159210 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 11:42:58.159212 (+ 2us) client_negotiation.cc:189] Negotiated authn=TOKEN
0620 11:42:58.159255 (+ 43us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 11:42:58.159256 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 11:42:58.159875 (+ 619us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 11:42:58.159876 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 11:42:58.160493 (+ 617us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 11:42:58.160495 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 11:42:58.160944 (+ 449us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 11:42:58.160945 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 11:42:58.160995 (+ 50us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 11:42:58.160999 (+ 4us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0620 11:42:58.161169 (+ 170us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0620 11:42:58.161171 (+ 2us) client_negotiation.cc:769] Sending connection context
0620 11:42:58.161228 (+ 57us) client_negotiation.cc:240] Negotiation successful
0620 11:42:58.161272 (+ 44us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":18}
I20260620 11:42:58.161586 15089 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 11:42:58.159018 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.6.234.194:35621 (local address 127.6.234.193:44461)
0620 11:42:58.159103 (+ 85us) server_negotiation.cc:206] Beginning negotiation
0620 11:42:58.159105 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 11:42:58.159113 (+ 8us) server_negotiation.cc:407] Connection header received
0620 11:42:58.159136 (+ 23us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 11:42:58.159137 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 11:42:58.159148 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 11:42:58.159207 (+ 59us) server_negotiation.cc:226] Negotiated authn=TOKEN
0620 11:42:58.159335 (+ 128us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 11:42:58.159779 (+ 444us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 11:42:58.160657 (+ 878us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 11:42:58.160872 (+ 215us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 11:42:58.160947 (+ 75us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 11:42:58.161079 (+ 132us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0620 11:42:58.161107 (+ 28us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0620 11:42:58.161148 (+ 41us) server_negotiation.cc:1035] Waiting for connection context
0620 11:42:58.161506 (+ 358us) server_negotiation.cc:299] Negotiation successful
0620 11:42:58.161527 (+ 21us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":61}
I20260620 11:42:58.167546 14583 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:46562:
table { table_name: "test-table" } modify_external_catalogs: true
I20260620 11:42:58.167655 14583 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:46562:
table { table_name: "test-table" } modify_external_catalogs: true
I20260620 11:42:58.168286 14583 catalog_manager.cc:5782] T 00000000000000000000000000000000 P f75e5ca955994ed9ba7f8b6ce11b9c5a: Sending DeleteTablet for 3 replicas of tablet 04ff4b4ec62e40d0a9cd57d29da28d20
I20260620 11:42:58.168512 14720 tablet_service.cc:1514] Processing DeleteTablet for tablet 04ff4b4ec62e40d0a9cd57d29da28d20 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 11:42:58 UTC) from {username='oryx', principal='oryx/127.6.234.254@KRBTEST.COM'} at 127.0.0.1:40862
I20260620 11:42:58.168566 15000 tablet_service.cc:1514] Processing DeleteTablet for tablet 04ff4b4ec62e40d0a9cd57d29da28d20 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 11:42:58 UTC) from {username='oryx', principal='oryx/127.6.234.254@KRBTEST.COM'} at 127.0.0.1:44454
I20260620 11:42:58.168567 14859 tablet_service.cc:1514] Processing DeleteTablet for tablet 04ff4b4ec62e40d0a9cd57d29da28d20 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 11:42:58 UTC) from {username='oryx', principal='oryx/127.6.234.254@KRBTEST.COM'} at 127.0.0.1:39656
I20260620 11:42:58.168607 15115 tablet_replica.cc:330] T 04ff4b4ec62e40d0a9cd57d29da28d20 P 580c848b417d4ffbab6d865896c13de7: stopping tablet replica
I20260620 11:42:58.168648 15114 tablet_replica.cc:330] T 04ff4b4ec62e40d0a9cd57d29da28d20 P d4e3cd899940498d83eba8fc5ed6b9a2: stopping tablet replica
I20260620 11:42:58.168653 15115 raft_consensus.cc:2229] T 04ff4b4ec62e40d0a9cd57d29da28d20 P 580c848b417d4ffbab6d865896c13de7 [term 1 LEADER]: Raft consensus shutting down.
I20260620 11:42:58.168708 15114 raft_consensus.cc:2229] T 04ff4b4ec62e40d0a9cd57d29da28d20 P d4e3cd899940498d83eba8fc5ed6b9a2 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260620 11:42:58.168720 15115 raft_consensus.cc:2258] T 04ff4b4ec62e40d0a9cd57d29da28d20 P 580c848b417d4ffbab6d865896c13de7 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 11:42:58.168745 15114 raft_consensus.cc:2258] T 04ff4b4ec62e40d0a9cd57d29da28d20 P d4e3cd899940498d83eba8fc5ed6b9a2 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 11:42:58.168812 15113 tablet_replica.cc:330] T 04ff4b4ec62e40d0a9cd57d29da28d20 P 82db96227ab34c6e93967a5951456933: stopping tablet replica
I20260620 11:42:58.168895 15113 raft_consensus.cc:2229] T 04ff4b4ec62e40d0a9cd57d29da28d20 P 82db96227ab34c6e93967a5951456933 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260620 11:42:58.168949 15113 raft_consensus.cc:2258] T 04ff4b4ec62e40d0a9cd57d29da28d20 P 82db96227ab34c6e93967a5951456933 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 11:42:58.169090 15114 ts_tablet_manager.cc:1905] T 04ff4b4ec62e40d0a9cd57d29da28d20 P d4e3cd899940498d83eba8fc5ed6b9a2: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 11:42:58.169203 15115 ts_tablet_manager.cc:1905] T 04ff4b4ec62e40d0a9cd57d29da28d20 P 580c848b417d4ffbab6d865896c13de7: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 11:42:58.169514 7083 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task1JRg7N/build/release/bin/kudu with pid 14655
I20260620 11:42:58.170789 15114 ts_tablet_manager.cc:1918] T 04ff4b4ec62e40d0a9cd57d29da28d20 P d4e3cd899940498d83eba8fc5ed6b9a2: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260620 11:42:58.170866 15114 log.cc:1196] T 04ff4b4ec62e40d0a9cd57d29da28d20 P d4e3cd899940498d83eba8fc5ed6b9a2: Deleting WAL directory at /tmp/dist-test-task1JRg7N/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781955746769051-7083-0/minicluster-data/ts-2/wal/wals/04ff4b4ec62e40d0a9cd57d29da28d20
I20260620 11:42:58.171094 15114 ts_tablet_manager.cc:1939] T 04ff4b4ec62e40d0a9cd57d29da28d20 P d4e3cd899940498d83eba8fc5ed6b9a2: Deleting consensus metadata
I20260620 11:42:58.171416 14569 catalog_manager.cc:4841] TS d4e3cd899940498d83eba8fc5ed6b9a2 (127.6.234.195:38309): tablet 04ff4b4ec62e40d0a9cd57d29da28d20 (table test-table [id=b0538bdf02bc41cda8d0a3a823c8e0ee]) successfully deleted
I20260620 11:42:58.171504 15113 ts_tablet_manager.cc:1905] T 04ff4b4ec62e40d0a9cd57d29da28d20 P 82db96227ab34c6e93967a5951456933: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 11:42:58.172695 15113 ts_tablet_manager.cc:1918] T 04ff4b4ec62e40d0a9cd57d29da28d20 P 82db96227ab34c6e93967a5951456933: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260620 11:42:58.172768 15113 log.cc:1196] T 04ff4b4ec62e40d0a9cd57d29da28d20 P 82db96227ab34c6e93967a5951456933: Deleting WAL directory at /tmp/dist-test-task1JRg7N/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781955746769051-7083-0/minicluster-data/ts-1/wal/wals/04ff4b4ec62e40d0a9cd57d29da28d20
I20260620 11:42:58.172981 15113 ts_tablet_manager.cc:1939] T 04ff4b4ec62e40d0a9cd57d29da28d20 P 82db96227ab34c6e93967a5951456933: Deleting consensus metadata
I20260620 11:42:58.173297 14570 catalog_manager.cc:4841] TS 82db96227ab34c6e93967a5951456933 (127.6.234.194:40195): tablet 04ff4b4ec62e40d0a9cd57d29da28d20 (table test-table [id=b0538bdf02bc41cda8d0a3a823c8e0ee]) successfully deleted
W20260620 11:42:58.176822 14567 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.6.234.193:44461 (error 108)
I20260620 11:42:58.177080 7083 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task1JRg7N/build/release/bin/kudu with pid 14795
I20260620 11:42:58.177871 15177 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 11:42:58.177227 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.234.193:44461 (local address 127.0.0.1:40886)
0620 11:42:58.177771 (+ 544us) negotiation.cc:107] Waiting for socket to connect
0620 11:42:58.177819 (+ 48us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.234.193:44461: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":524,"thread_start_us":43,"threads_started":1}
W20260620 11:42:58.178012 14567 catalog_manager.cc:4584] TS 580c848b417d4ffbab6d865896c13de7 (127.6.234.193:44461): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 04ff4b4ec62e40d0a9cd57d29da28d20: Network error: Client connection negotiation failed: client connection to 127.6.234.193:44461: connect: Connection refused (error 111)
I20260620 11:42:58.183580 7083 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task1JRg7N/build/release/bin/kudu with pid 14935
I20260620 11:42:58.188946 7083 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task1JRg7N/build/release/bin/kudu with pid 14551
2026-06-20T11:42:58Z chronyd exiting
[ OK ] SecurityITest.TestNonDefaultPrincipal (2433 ms)
[ RUN ] SecurityITest.TestNonExistentPrincipal
Loading random data
Initializing database '/tmp/dist-test-task1JRg7N/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1781955746769051-7083-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 20 11:42:58 dist-test-slave-cjc6 krb5kdc[15180](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 20 11:42:58 dist-test-slave-cjc6 krb5kdc[15180](info): set up 2 sockets
Jun 20 11:42:58 dist-test-slave-cjc6 krb5kdc[15180](info): commencing operation
krb5kdc: starting...
W20260620 11:42:59.227697 7083 mini_kdc.cc:121] Time spent starting KDC: real 1.018s user 0.000s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task1JRg7N/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1781955746769051-7083-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task1JRg7N/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1781955746769051-7083-0/krb5kdc/test-user.keytab.
Jun 20 11:42:59 dist-test-slave-cjc6 krb5kdc[15180](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781955779, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-20T11:42:59Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-20T11:42:59Z Disabled control of system clock
WARNING: no policy specified for kudu/127.6.234.254@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.6.234.254@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.6.234.254 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task1JRg7N/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1781955746769051-7083-0/krb5kdc/kudu_127.6.234.254.keytab.
Entry for principal kudu/127.6.234.254 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task1JRg7N/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1781955746769051-7083-0/krb5kdc/kudu_127.6.234.254.keytab.
I20260620 11:42:59.348335 7083 external_mini_cluster.cc:1326] Running /tmp/dist-test-task1JRg7N/build/release/bin/kudu
/tmp/dist-test-task1JRg7N/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task1JRg7N/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1781955746769051-7083-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task1JRg7N/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1781955746769051-7083-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task1JRg7N/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1781955746769051-7083-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task1JRg7N/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1781955746769051-7083-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.6.234.254:35211
--webserver_interface=127.6.234.254
--webserver_port=0
--keytab_file=/tmp/dist-test-task1JRg7N/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1781955746769051-7083-0/krb5kdc/kudu_127.6.234.254.keytab
--principal=kudu/127.6.234.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.6.234.212:37097
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.6.234.254:35211
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--principal=oryx with env {KRB5CCNAME=/tmp/dist-test-task1JRg7N/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1781955746769051-7083-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task1JRg7N/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1781955746769051-7083-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task1JRg7N/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1781955746769051-7083-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 11:42:59.427707 15193 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 11:42:59.427975 15193 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 11:42:59.428017 15193 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 11:42:59.429281 15193 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 11:42:59.429324 15193 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 11:42:59.429338 15193 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 11:42:59.429352 15193 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 11:42:59.429365 15193 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 11:42:59.430922 15193 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.6.234.212:37097
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task1JRg7N/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1781955746769051-7083-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task1JRg7N/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1781955746769051-7083-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.6.234.254:35211
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task1JRg7N/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1781955746769051-7083-0/krb5kdc/kudu_127.6.234.254.keytab
--principal=oryx
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.6.234.254:35211
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task1JRg7N/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1781955746769051-7083-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.6.234.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.15193
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task1JRg7N/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1781955746769051-7083-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 20 Jun 2026 10:32:37 UTC on ccycloud.quasar-bgyofv.root.comops.site
build id 4457173
I20260620 11:42:59.431169 15193 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 11:42:59.431389 15193 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 11:42:59.436210 15201 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 11:42:59.436224 15198 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 11:42:59.436233 15199 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 11:42:59.436407 15193 server_base.cc:1034] running on GCE node
I20260620 11:42:59.436700 15193 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 11:42:59.436988 15193 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 11:42:59.438169 15193 hybrid_clock.cc:648] HybridClock initialized: now 1781955779438157 us; error 41 us; skew 500 ppm
Runtime error: kinit failed: unable to login from keytab: Keytab contains no suitable keys for oryx@KRBTEST.COM
2026-06-20T11:42:59Z chronyd exiting
[ OK ] SecurityITest.TestNonExistentPrincipal (1256 ms)
[ RUN ] SecurityITest.TestMismatchingPrincipals
Loading random data
Initializing database '/tmp/dist-test-task1JRg7N/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781955746769051-7083-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 20 11:42:59 dist-test-slave-cjc6 krb5kdc[15206](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 20 11:42:59 dist-test-slave-cjc6 krb5kdc[15206](info): set up 2 sockets
Jun 20 11:42:59 dist-test-slave-cjc6 krb5kdc[15206](info): commencing operation
krb5kdc: starting...
W20260620 11:43:00.482410 7083 mini_kdc.cc:121] Time spent starting KDC: real 1.017s user 0.000s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task1JRg7N/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781955746769051-7083-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task1JRg7N/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781955746769051-7083-0/krb5kdc/test-user.keytab.
Jun 20 11:43:00 dist-test-slave-cjc6 krb5kdc[15206](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781955780, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-20T11:43:00Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-20T11:43:00Z Disabled control of system clock
WARNING: no policy specified for kudu/127.6.234.254@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.6.234.254@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.6.234.254 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task1JRg7N/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781955746769051-7083-0/krb5kdc/kudu_127.6.234.254.keytab.
Entry for principal kudu/127.6.234.254 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task1JRg7N/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781955746769051-7083-0/krb5kdc/kudu_127.6.234.254.keytab.
I20260620 11:43:00.598670 7083 external_mini_cluster.cc:1326] Running /tmp/dist-test-task1JRg7N/build/release/bin/kudu
/tmp/dist-test-task1JRg7N/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task1JRg7N/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781955746769051-7083-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task1JRg7N/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781955746769051-7083-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task1JRg7N/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781955746769051-7083-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task1JRg7N/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781955746769051-7083-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.6.234.254:38223
--webserver_interface=127.6.234.254
--webserver_port=0
--keytab_file=/tmp/dist-test-task1JRg7N/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781955746769051-7083-0/krb5kdc/kudu_127.6.234.254.keytab
--principal=kudu/127.6.234.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.6.234.212:40309
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.6.234.254:38223
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task1JRg7N/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781955746769051-7083-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task1JRg7N/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781955746769051-7083-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task1JRg7N/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781955746769051-7083-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 11:43:00.679188 15219 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 11:43:00.679459 15219 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 11:43:00.679497 15219 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 11:43:00.680619 15219 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 11:43:00.680691 15219 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 11:43:00.680713 15219 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 11:43:00.680735 15219 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 11:43:00.680766 15219 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 11:43:00.682225 15219 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.6.234.212:40309
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task1JRg7N/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781955746769051-7083-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task1JRg7N/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781955746769051-7083-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.6.234.254:38223
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task1JRg7N/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781955746769051-7083-0/krb5kdc/kudu_127.6.234.254.keytab
--principal=kudu/127.6.234.254
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.6.234.254:38223
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task1JRg7N/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781955746769051-7083-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.6.234.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.15219
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task1JRg7N/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781955746769051-7083-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 20 Jun 2026 10:32:37 UTC on ccycloud.quasar-bgyofv.root.comops.site
build id 4457173
I20260620 11:43:00.682502 15219 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 11:43:00.682729 15219 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260620 11:43:00.687618 15219 server_base.cc:1034] running on GCE node
W20260620 11:43:00.687595 15225 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 11:43:00.687583 15227 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 11:43:00.687583 15224 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 11:43:00.687991 15219 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 11:43:00.688269 15219 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 11:43:00.689440 15219 hybrid_clock.cc:648] HybridClock initialized: now 1781955780689437 us; error 37 us; skew 500 ppm
Jun 20 11:43:00 dist-test-slave-cjc6 krb5kdc[15206](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781955780, etypes {rep=17 tkt=17 ses=17}, kudu/127.6.234.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 11:43:00.691926 15219 init.cc:377] Logged in from keytab as kudu/127.6.234.254@KRBTEST.COM (short username kudu)
I20260620 11:43:00.692350 15219 webserver.cc:460] Webserver started at http://127.6.234.254:41159/ using document root <none> and password file <none>
I20260620 11:43:00.692591 15219 fs_manager.cc:359] Metadata directory not provided
I20260620 11:43:00.692669 15219 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 11:43:00.692785 15219 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 11:43:00.693858 15219 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task1JRg7N/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781955746769051-7083-0/minicluster-data/master-0/data/instance:
uuid: "9505661ea1f54dcbb8c78592e5491859"
format_stamp: "Formatted at 2026-06-20 11:43:00 on dist-test-slave-cjc6"
server_key: "27b6cc1fead2a7a7b64f721e86190630"
server_key_iv: "ea3e840295d2223970c4cff32110f6f0"
server_key_version: "encryptionkey@0"
I20260620 11:43:00.694146 15219 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task1JRg7N/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781955746769051-7083-0/minicluster-data/master-0/wal/instance:
uuid: "9505661ea1f54dcbb8c78592e5491859"
format_stamp: "Formatted at 2026-06-20 11:43:00 on dist-test-slave-cjc6"
server_key: "27b6cc1fead2a7a7b64f721e86190630"
server_key_iv: "ea3e840295d2223970c4cff32110f6f0"
server_key_version: "encryptionkey@0"
I20260620 11:43:00.695391 15219 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.001s
I20260620 11:43:00.696051 15234 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 11:43:00.696230 15219 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.001s
I20260620 11:43:00.696327 15219 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task1JRg7N/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781955746769051-7083-0/minicluster-data/master-0/data,/tmp/dist-test-task1JRg7N/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781955746769051-7083-0/minicluster-data/master-0/wal
uuid: "9505661ea1f54dcbb8c78592e5491859"
format_stamp: "Formatted at 2026-06-20 11:43:00 on dist-test-slave-cjc6"
server_key: "27b6cc1fead2a7a7b64f721e86190630"
server_key_iv: "ea3e840295d2223970c4cff32110f6f0"
server_key_version: "encryptionkey@0"
I20260620 11:43:00.696410 15219 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task1JRg7N/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781955746769051-7083-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task1JRg7N/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781955746769051-7083-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task1JRg7N/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781955746769051-7083-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 11:43:00.706538 15219 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 11:43:00.708300 15219 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 11:43:00.708454 15219 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 11:43:00.712391 15219 rpc_server.cc:307] RPC server started. Bound to: 127.6.234.254:38223
I20260620 11:43:00.712427 15286 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.6.234.254:38223 every 8 connection(s)
I20260620 11:43:00.712803 15219 server_base.cc:1166] Dumped server information to /tmp/dist-test-task1JRg7N/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781955746769051-7083-0/minicluster-data/master-0/data/info.pb
I20260620 11:43:00.713054 7083 external_mini_cluster.cc:1388] Started /tmp/dist-test-task1JRg7N/build/release/bin/kudu as pid 15219
I20260620 11:43:00.713207 7083 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task1JRg7N/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781955746769051-7083-0/minicluster-data/master-0/wal/instance
I20260620 11:43:00.713430 7083 external_mini_cluster.cc:1428] Setting key 0d9ce635c0f88d8d9c655834ac332c1a
I20260620 11:43:00.713809 15287 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 11:43:00.716548 15287 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 9505661ea1f54dcbb8c78592e5491859: Bootstrap starting.
I20260620 11:43:00.717171 15287 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 9505661ea1f54dcbb8c78592e5491859: Neither blocks nor log segments found. Creating new log.
I20260620 11:43:00.717461 15287 log.cc:824] T 00000000000000000000000000000000 P 9505661ea1f54dcbb8c78592e5491859: Log is configured to *not* fsync() on all Append() calls
I20260620 11:43:00.718271 15287 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 9505661ea1f54dcbb8c78592e5491859: No bootstrap required, opened a new log
I20260620 11:43:00.720209 15287 raft_consensus.cc:348] T 00000000000000000000000000000000 P 9505661ea1f54dcbb8c78592e5491859 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9505661ea1f54dcbb8c78592e5491859" member_type: VOTER last_known_addr { host: "127.6.234.254" port: 38223 } }
I20260620 11:43:00.720352 15287 raft_consensus.cc:374] T 00000000000000000000000000000000 P 9505661ea1f54dcbb8c78592e5491859 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 11:43:00.720376 15287 raft_consensus.cc:729] T 00000000000000000000000000000000 P 9505661ea1f54dcbb8c78592e5491859 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9505661ea1f54dcbb8c78592e5491859, State: Initialized, Role: FOLLOWER
I20260620 11:43:00.720512 15287 consensus_queue.cc:260] T 00000000000000000000000000000000 P 9505661ea1f54dcbb8c78592e5491859 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9505661ea1f54dcbb8c78592e5491859" member_type: VOTER last_known_addr { host: "127.6.234.254" port: 38223 } }
I20260620 11:43:00.720592 15287 raft_consensus.cc:388] T 00000000000000000000000000000000 P 9505661ea1f54dcbb8c78592e5491859 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260620 11:43:00.720614 15287 raft_consensus.cc:482] T 00000000000000000000000000000000 P 9505661ea1f54dcbb8c78592e5491859 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260620 11:43:00.720679 15287 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 9505661ea1f54dcbb8c78592e5491859 [term 0 FOLLOWER]: Advancing to term 1
Jun 20 11:43:00 dist-test-slave-cjc6 krb5kdc[15206](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781955780, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.6.234.254@KRBTEST.COM
I20260620 11:43:00.721262 15287 raft_consensus.cc:504] T 00000000000000000000000000000000 P 9505661ea1f54dcbb8c78592e5491859 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9505661ea1f54dcbb8c78592e5491859" member_type: VOTER last_known_addr { host: "127.6.234.254" port: 38223 } }
I20260620 11:43:00.721400 15287 leader_election.cc:302] T 00000000000000000000000000000000 P 9505661ea1f54dcbb8c78592e5491859 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 9505661ea1f54dcbb8c78592e5491859; no voters:
I20260620 11:43:00.721613 15287 leader_election.cc:288] T 00000000000000000000000000000000 P 9505661ea1f54dcbb8c78592e5491859 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260620 11:43:00.721725 15292 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 9505661ea1f54dcbb8c78592e5491859 [term 1 FOLLOWER]: Leader election won for term 1
I20260620 11:43:00.721899 15292 raft_consensus.cc:686] T 00000000000000000000000000000000 P 9505661ea1f54dcbb8c78592e5491859 [term 1 LEADER]: Becoming Leader. State: Replica: 9505661ea1f54dcbb8c78592e5491859, State: Running, Role: LEADER
I20260620 11:43:00.721899 15287 sys_catalog.cc:564] T 00000000000000000000000000000000 P 9505661ea1f54dcbb8c78592e5491859 [sys.catalog]: configured and running, proceeding with master startup.
I20260620 11:43:00.722014 15292 consensus_queue.cc:237] T 00000000000000000000000000000000 P 9505661ea1f54dcbb8c78592e5491859 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9505661ea1f54dcbb8c78592e5491859" member_type: VOTER last_known_addr { host: "127.6.234.254" port: 38223 } }
I20260620 11:43:00.722399 15293 sys_catalog.cc:455] T 00000000000000000000000000000000 P 9505661ea1f54dcbb8c78592e5491859 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "9505661ea1f54dcbb8c78592e5491859" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9505661ea1f54dcbb8c78592e5491859" member_type: VOTER last_known_addr { host: "127.6.234.254" port: 38223 } } }
I20260620 11:43:00.722675 15294 sys_catalog.cc:455] T 00000000000000000000000000000000 P 9505661ea1f54dcbb8c78592e5491859 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 9505661ea1f54dcbb8c78592e5491859. Latest consensus state: current_term: 1 leader_uuid: "9505661ea1f54dcbb8c78592e5491859" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9505661ea1f54dcbb8c78592e5491859" member_type: VOTER last_known_addr { host: "127.6.234.254" port: 38223 } } }
I20260620 11:43:00.722826 15294 sys_catalog.cc:458] T 00000000000000000000000000000000 P 9505661ea1f54dcbb8c78592e5491859 [sys.catalog]: This master's current role is: LEADER
I20260620 11:43:00.723152 15303 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260620 11:43:00.723639 15303 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260620 11:43:00.722550 15293 sys_catalog.cc:458] T 00000000000000000000000000000000 P 9505661ea1f54dcbb8c78592e5491859 [sys.catalog]: This master's current role is: LEADER
I20260620 11:43:00.725564 15290 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 11:43:00.714050 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40126 (local address 127.6.234.254:38223)
0620 11:43:00.714249 (+ 199us) server_negotiation.cc:206] Beginning negotiation
0620 11:43:00.714259 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0620 11:43:00.714300 (+ 41us) server_negotiation.cc:407] Connection header received
0620 11:43:00.714950 (+ 650us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 11:43:00.714957 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 11:43:00.715008 (+ 51us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 11:43:00.715173 (+ 165us) server_negotiation.cc:226] Negotiated authn=SASL
0620 11:43:00.715850 (+ 677us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 11:43:00.716927 (+ 1077us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 11:43:00.717988 (+ 1061us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 11:43:00.718582 (+ 594us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 11:43:00.718730 (+ 148us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 11:43:00.721370 (+ 2640us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 11:43:00.721375 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 11:43:00.721375 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 11:43:00.721386 (+ 11us) server_negotiation.cc:922] calling sasl_server_start()
0620 11:43:00.723677 (+ 2291us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 11:43:00.724162 (+ 485us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 11:43:00.724166 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 11:43:00.724166 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 11:43:00.724278 (+ 112us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 11:43:00.724516 (+ 238us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 11:43:00.724517 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 11:43:00.724518 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 11:43:00.724860 (+ 342us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 11:43:00.725024 (+ 164us) server_negotiation.cc:1035] Waiting for connection context
0620 11:43:00.725186 (+ 162us) server_negotiation.cc:299] Negotiation successful
0620 11:43:00.725351 (+ 165us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":144,"thread_start_us":86,"threads_started":1}
I20260620 11:43:00.726084 15303 catalog_manager.cc:1347] Generated new cluster ID: 74e1ace3749840ab977b124557aa3e82
I20260620 11:43:00.726137 15303 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260620 11:43:00.731711 15303 catalog_manager.cc:1370] Generated new certificate authority record
I20260620 11:43:00.732286 15303 catalog_manager.cc:1504] Loading token signing keys...
I20260620 11:43:00.743988 15303 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 9505661ea1f54dcbb8c78592e5491859: Generated new TSK 0
I20260620 11:43:00.744205 15303 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.6.234.193@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.6.234.193@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.6.234.193 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task1JRg7N/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781955746769051-7083-0/krb5kdc/kudu_127.6.234.193.keytab.
Entry for principal kudu/127.6.234.193 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task1JRg7N/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781955746769051-7083-0/krb5kdc/kudu_127.6.234.193.keytab.
I20260620 11:43:00.772243 7083 external_mini_cluster.cc:1326] Running /tmp/dist-test-task1JRg7N/build/release/bin/kudu
/tmp/dist-test-task1JRg7N/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task1JRg7N/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781955746769051-7083-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-task1JRg7N/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781955746769051-7083-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task1JRg7N/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781955746769051-7083-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-task1JRg7N/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781955746769051-7083-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.6.234.193:0
--local_ip_for_outbound_sockets=127.6.234.193
--webserver_interface=127.6.234.193
--webserver_port=0
--tserver_master_addrs=127.6.234.254:38223
--keytab_file=/tmp/dist-test-task1JRg7N/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781955746769051-7083-0/krb5kdc/kudu_127.6.234.193.keytab
--principal=kudu/127.6.234.193
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.6.234.212:40309
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task1JRg7N/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781955746769051-7083-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task1JRg7N/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781955746769051-7083-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task1JRg7N/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781955746769051-7083-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 11:43:00.867986 15313 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 11:43:00.868196 15313 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 11:43:00.868222 15313 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 11:43:00.869429 15313 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 11:43:00.869475 15313 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 11:43:00.869503 15313 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.6.234.193
I20260620 11:43:00.871032 15313 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.6.234.212:40309
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task1JRg7N/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781955746769051-7083-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-task1JRg7N/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781955746769051-7083-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task1JRg7N/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781955746769051-7083-0/krb5kdc/kudu_127.6.234.193.keytab
--principal=kudu/127.6.234.193
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.6.234.193:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task1JRg7N/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781955746769051-7083-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.6.234.193
--webserver_port=0
--tserver_master_addrs=127.6.234.254:38223
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.15313
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.6.234.193
--log_dir=/tmp/dist-test-task1JRg7N/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781955746769051-7083-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 20 Jun 2026 10:32:37 UTC on ccycloud.quasar-bgyofv.root.comops.site
build id 4457173
I20260620 11:43:00.871287 15313 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 11:43:00.871543 15313 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 11:43:00.876451 15321 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 11:43:00.876518 15319 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 11:43:00.876734 15313 server_base.cc:1034] running on GCE node
W20260620 11:43:00.876515 15318 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 11:43:00.877022 15313 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 11:43:00.877235 15313 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 11:43:00.878388 15313 hybrid_clock.cc:648] HybridClock initialized: now 1781955780878376 us; error 28 us; skew 500 ppm
Jun 20 11:43:00 dist-test-slave-cjc6 krb5kdc[15206](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781955780, etypes {rep=17 tkt=17 ses=17}, kudu/127.6.234.193@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 11:43:00.880937 15313 init.cc:377] Logged in from keytab as kudu/127.6.234.193@KRBTEST.COM (short username kudu)
I20260620 11:43:00.881378 15313 webserver.cc:460] Webserver started at http://127.6.234.193:42849/ using document root <none> and password file <none>
I20260620 11:43:00.881649 15313 fs_manager.cc:359] Metadata directory not provided
I20260620 11:43:00.881739 15313 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 11:43:00.881916 15313 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 11:43:00.882972 15313 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task1JRg7N/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781955746769051-7083-0/minicluster-data/ts-0/data/instance:
uuid: "6e169ae0a436470fb4a87f555360e93e"
format_stamp: "Formatted at 2026-06-20 11:43:00 on dist-test-slave-cjc6"
server_key: "237554ec3b69d3aa0f49551f5b435622"
server_key_iv: "05cdb51a5d34be71ae7e73d2948be6a2"
server_key_version: "encryptionkey@0"
I20260620 11:43:00.883329 15313 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task1JRg7N/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781955746769051-7083-0/minicluster-data/ts-0/wal/instance:
uuid: "6e169ae0a436470fb4a87f555360e93e"
format_stamp: "Formatted at 2026-06-20 11:43:00 on dist-test-slave-cjc6"
server_key: "237554ec3b69d3aa0f49551f5b435622"
server_key_iv: "05cdb51a5d34be71ae7e73d2948be6a2"
server_key_version: "encryptionkey@0"
I20260620 11:43:00.884713 15313 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260620 11:43:00.885517 15328 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 11:43:00.885776 15313 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260620 11:43:00.885882 15313 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task1JRg7N/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781955746769051-7083-0/minicluster-data/ts-0/data,/tmp/dist-test-task1JRg7N/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781955746769051-7083-0/minicluster-data/ts-0/wal
uuid: "6e169ae0a436470fb4a87f555360e93e"
format_stamp: "Formatted at 2026-06-20 11:43:00 on dist-test-slave-cjc6"
server_key: "237554ec3b69d3aa0f49551f5b435622"
server_key_iv: "05cdb51a5d34be71ae7e73d2948be6a2"
server_key_version: "encryptionkey@0"
I20260620 11:43:00.885962 15313 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task1JRg7N/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781955746769051-7083-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-task1JRg7N/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781955746769051-7083-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-task1JRg7N/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781955746769051-7083-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 11:43:00.898432 15313 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 11:43:00.900995 15313 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 11:43:00.901154 15313 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 11:43:00.901393 15313 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 11:43:00.901746 15313 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 11:43:00.901805 15313 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 11:43:00.901840 15313 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 11:43:00.901868 15313 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 11:43:00.908154 15313 rpc_server.cc:307] RPC server started. Bound to: 127.6.234.193:38929
I20260620 11:43:00.908205 15441 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.6.234.193:38929 every 8 connection(s)
I20260620 11:43:00.908522 15313 server_base.cc:1166] Dumped server information to /tmp/dist-test-task1JRg7N/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781955746769051-7083-0/minicluster-data/ts-0/data/info.pb
Jun 20 11:43:00 dist-test-slave-cjc6 krb5kdc[15206](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781955780, etypes {rep=17 tkt=17 ses=17}, kudu/127.6.234.193@KRBTEST.COM for kudu/127.6.234.254@KRBTEST.COM
I20260620 11:43:00.917533 15290 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 11:43:00.909317 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.6.234.193:50553 (local address 127.6.234.254:38223)
0620 11:43:00.909381 (+ 64us) server_negotiation.cc:206] Beginning negotiation
0620 11:43:00.909383 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 11:43:00.909541 (+ 158us) server_negotiation.cc:407] Connection header received
0620 11:43:00.910147 (+ 606us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 11:43:00.910157 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 11:43:00.910167 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 11:43:00.910240 (+ 73us) server_negotiation.cc:226] Negotiated authn=SASL
0620 11:43:00.911171 (+ 931us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 11:43:00.911668 (+ 497us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 11:43:00.912646 (+ 978us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 11:43:00.912856 (+ 210us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 11:43:00.912913 (+ 57us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 11:43:00.915086 (+ 2173us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 11:43:00.915087 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 11:43:00.915088 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 11:43:00.915094 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0620 11:43:00.916293 (+ 1199us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 11:43:00.916677 (+ 384us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 11:43:00.916678 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 11:43:00.916678 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 11:43:00.916726 (+ 48us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 11:43:00.916997 (+ 271us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 11:43:00.916997 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 11:43:00.916997 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 11:43:00.917100 (+ 103us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 11:43:00.917149 (+ 49us) server_negotiation.cc:1035] Waiting for connection context
0620 11:43:00.917397 (+ 248us) server_negotiation.cc:299] Negotiation successful
0620 11:43:00.917466 (+ 69us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":38}
I20260620 11:43:00.917752 15444 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 11:43:00.909257 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.234.254:38223 (local address 127.6.234.193:50553)
0620 11:43:00.909411 (+ 154us) negotiation.cc:107] Waiting for socket to connect
0620 11:43:00.909453 (+ 42us) client_negotiation.cc:174] Beginning negotiation
0620 11:43:00.910021 (+ 568us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 11:43:00.910321 (+ 300us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 11:43:00.910325 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 11:43:00.910889 (+ 564us) client_negotiation.cc:189] Negotiated authn=SASL
0620 11:43:00.911084 (+ 195us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 11:43:00.911089 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 11:43:00.911748 (+ 659us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 11:43:00.911749 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 11:43:00.912564 (+ 815us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 11:43:00.912565 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 11:43:00.912929 (+ 364us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 11:43:00.912930 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 11:43:00.913029 (+ 99us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 11:43:00.913514 (+ 485us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 11:43:00.913523 (+ 9us) client_negotiation.cc:656] calling sasl_client_start()
0620 11:43:00.914995 (+ 1472us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 11:43:00.916368 (+ 1373us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 11:43:00.916374 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 11:43:00.916376 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0620 11:43:00.916601 (+ 225us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 11:43:00.916806 (+ 205us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 11:43:00.916807 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 11:43:00.916807 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0620 11:43:00.916930 (+ 123us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 11:43:00.917155 (+ 225us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 11:43:00.917159 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 11:43:00.917283 (+ 124us) client_negotiation.cc:769] Sending connection context
0620 11:43:00.917370 (+ 87us) client_negotiation.cc:240] Negotiation successful
0620 11:43:00.917505 (+ 135us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":110,"thread_start_us":82,"threads_started":1}
I20260620 11:43:00.917922 7083 external_mini_cluster.cc:1388] Started /tmp/dist-test-task1JRg7N/build/release/bin/kudu as pid 15313
I20260620 11:43:00.918040 7083 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task1JRg7N/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781955746769051-7083-0/minicluster-data/ts-0/wal/instance
I20260620 11:43:00.918251 7083 external_mini_cluster.cc:1428] Setting key 095f7ec61143f98025637f3571697c08
I20260620 11:43:00.918428 15442 heartbeater.cc:347] Connected to a master server at 127.6.234.254:38223
I20260620 11:43:00.918551 15442 heartbeater.cc:464] Registering TS with master...
I20260620 11:43:00.918751 15442 heartbeater.cc:511] Master 127.6.234.254:38223 requested a full tablet report, sending...
I20260620 11:43:00.919350 15251 ts_manager.cc:194] Registered new tserver with Master: 6e169ae0a436470fb4a87f555360e93e (127.6.234.193:38929)
I20260620 11:43:00.920250 15251 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.6.234.193@KRBTEST.COM'} at 127.6.234.193:50553
WARNING: no policy specified for kudu/127.6.234.194@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.6.234.194@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.6.234.194 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task1JRg7N/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781955746769051-7083-0/krb5kdc/kudu_127.6.234.194.keytab.
Entry for principal kudu/127.6.234.194 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task1JRg7N/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781955746769051-7083-0/krb5kdc/kudu_127.6.234.194.keytab.
I20260620 11:43:00.944111 7083 external_mini_cluster.cc:1326] Running /tmp/dist-test-task1JRg7N/build/release/bin/kudu
/tmp/dist-test-task1JRg7N/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task1JRg7N/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781955746769051-7083-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-task1JRg7N/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781955746769051-7083-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task1JRg7N/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781955746769051-7083-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-task1JRg7N/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781955746769051-7083-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.6.234.194:0
--local_ip_for_outbound_sockets=127.6.234.194
--webserver_interface=127.6.234.194
--webserver_port=0
--tserver_master_addrs=127.6.234.254:38223
--keytab_file=/tmp/dist-test-task1JRg7N/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781955746769051-7083-0/krb5kdc/kudu_127.6.234.194.keytab
--principal=kudu/127.6.234.194
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.6.234.212:40309
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task1JRg7N/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781955746769051-7083-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task1JRg7N/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781955746769051-7083-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task1JRg7N/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781955746769051-7083-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 11:43:01.026451 15447 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 11:43:01.026666 15447 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 11:43:01.026690 15447 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 11:43:01.027848 15447 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 11:43:01.027894 15447 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 11:43:01.027920 15447 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.6.234.194
I20260620 11:43:01.029451 15447 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.6.234.212:40309
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task1JRg7N/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781955746769051-7083-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-task1JRg7N/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781955746769051-7083-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task1JRg7N/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781955746769051-7083-0/krb5kdc/kudu_127.6.234.194.keytab
--principal=kudu/127.6.234.194
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.6.234.194:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task1JRg7N/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781955746769051-7083-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.6.234.194
--webserver_port=0
--tserver_master_addrs=127.6.234.254:38223
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.15447
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.6.234.194
--log_dir=/tmp/dist-test-task1JRg7N/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781955746769051-7083-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 20 Jun 2026 10:32:37 UTC on ccycloud.quasar-bgyofv.root.comops.site
build id 4457173
I20260620 11:43:01.029644 15447 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 11:43:01.029891 15447 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 11:43:01.034809 15453 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 11:43:01.034833 15455 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 11:43:01.034879 15452 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 11:43:01.034951 15447 server_base.cc:1034] running on GCE node
I20260620 11:43:01.035233 15447 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 11:43:01.035485 15447 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 11:43:01.036664 15447 hybrid_clock.cc:648] HybridClock initialized: now 1781955781036654 us; error 36 us; skew 500 ppm
Jun 20 11:43:01 dist-test-slave-cjc6 krb5kdc[15206](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781955781, etypes {rep=17 tkt=17 ses=17}, kudu/127.6.234.194@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 11:43:01.039384 15447 init.cc:377] Logged in from keytab as kudu/127.6.234.194@KRBTEST.COM (short username kudu)
I20260620 11:43:01.039845 15447 webserver.cc:460] Webserver started at http://127.6.234.194:40853/ using document root <none> and password file <none>
I20260620 11:43:01.040094 15447 fs_manager.cc:359] Metadata directory not provided
I20260620 11:43:01.040140 15447 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 11:43:01.040282 15447 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 11:43:01.041345 15447 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task1JRg7N/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781955746769051-7083-0/minicluster-data/ts-1/data/instance:
uuid: "98fae7d958fb4725b75f7b70e8b8d690"
format_stamp: "Formatted at 2026-06-20 11:43:01 on dist-test-slave-cjc6"
server_key: "8420df0ab3929d45dbdfbfb2cc8650b2"
server_key_iv: "02d03e53234502c7c66c3159c63cfbfe"
server_key_version: "encryptionkey@0"
I20260620 11:43:01.041751 15447 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task1JRg7N/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781955746769051-7083-0/minicluster-data/ts-1/wal/instance:
uuid: "98fae7d958fb4725b75f7b70e8b8d690"
format_stamp: "Formatted at 2026-06-20 11:43:01 on dist-test-slave-cjc6"
server_key: "8420df0ab3929d45dbdfbfb2cc8650b2"
server_key_iv: "02d03e53234502c7c66c3159c63cfbfe"
server_key_version: "encryptionkey@0"
I20260620 11:43:01.043105 15447 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.001s
I20260620 11:43:01.043869 15462 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 11:43:01.044085 15447 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.000s
I20260620 11:43:01.044203 15447 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task1JRg7N/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781955746769051-7083-0/minicluster-data/ts-1/data,/tmp/dist-test-task1JRg7N/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781955746769051-7083-0/minicluster-data/ts-1/wal
uuid: "98fae7d958fb4725b75f7b70e8b8d690"
format_stamp: "Formatted at 2026-06-20 11:43:01 on dist-test-slave-cjc6"
server_key: "8420df0ab3929d45dbdfbfb2cc8650b2"
server_key_iv: "02d03e53234502c7c66c3159c63cfbfe"
server_key_version: "encryptionkey@0"
I20260620 11:43:01.044292 15447 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task1JRg7N/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781955746769051-7083-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-task1JRg7N/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781955746769051-7083-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-task1JRg7N/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781955746769051-7083-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 11:43:01.063233 15447 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 11:43:01.065448 15447 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 11:43:01.065616 15447 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 11:43:01.065891 15447 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 11:43:01.066210 15447 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 11:43:01.066262 15447 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 11:43:01.066309 15447 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 11:43:01.066340 15447 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 11:43:01.072063 15447 rpc_server.cc:307] RPC server started. Bound to: 127.6.234.194:45005
I20260620 11:43:01.072142 15575 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.6.234.194:45005 every 8 connection(s)
I20260620 11:43:01.072463 15447 server_base.cc:1166] Dumped server information to /tmp/dist-test-task1JRg7N/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781955746769051-7083-0/minicluster-data/ts-1/data/info.pb
Jun 20 11:43:01 dist-test-slave-cjc6 krb5kdc[15206](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781955781, etypes {rep=17 tkt=17 ses=17}, kudu/127.6.234.194@KRBTEST.COM for kudu/127.6.234.254@KRBTEST.COM
I20260620 11:43:01.079428 7083 external_mini_cluster.cc:1388] Started /tmp/dist-test-task1JRg7N/build/release/bin/kudu as pid 15447
I20260620 11:43:01.079550 7083 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task1JRg7N/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781955746769051-7083-0/minicluster-data/ts-1/wal/instance
I20260620 11:43:01.079820 7083 external_mini_cluster.cc:1428] Setting key ae0af52099b8b76ff1f59598e6ac7a98
I20260620 11:43:01.082552 15578 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 11:43:01.073269 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.234.254:38223 (local address 127.6.234.194:50199)
0620 11:43:01.073438 (+ 169us) negotiation.cc:107] Waiting for socket to connect
0620 11:43:01.073480 (+ 42us) client_negotiation.cc:174] Beginning negotiation
0620 11:43:01.074014 (+ 534us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 11:43:01.074226 (+ 212us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 11:43:01.074229 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 11:43:01.074777 (+ 548us) client_negotiation.cc:189] Negotiated authn=SASL
0620 11:43:01.074968 (+ 191us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 11:43:01.074972 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 11:43:01.075615 (+ 643us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 11:43:01.075615 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 11:43:01.076409 (+ 794us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 11:43:01.076411 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 11:43:01.076855 (+ 444us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 11:43:01.076856 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 11:43:01.076974 (+ 118us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 11:43:01.077479 (+ 505us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 11:43:01.077488 (+ 9us) client_negotiation.cc:656] calling sasl_client_start()
0620 11:43:01.078950 (+ 1462us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 11:43:01.080877 (+ 1927us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 11:43:01.080884 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 11:43:01.080886 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0620 11:43:01.081223 (+ 337us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 11:43:01.081447 (+ 224us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 11:43:01.081448 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 11:43:01.081448 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0620 11:43:01.081593 (+ 145us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 11:43:01.081918 (+ 325us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 11:43:01.081925 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 11:43:01.082101 (+ 176us) client_negotiation.cc:769] Sending connection context
0620 11:43:01.082243 (+ 142us) client_negotiation.cc:240] Negotiation successful
0620 11:43:01.082390 (+ 147us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":130,"thread_start_us":94,"threads_started":1}
I20260620 11:43:01.082553 15290 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 11:43:01.073276 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.6.234.194:50199 (local address 127.6.234.254:38223)
0620 11:43:01.073343 (+ 67us) server_negotiation.cc:206] Beginning negotiation
0620 11:43:01.073345 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 11:43:01.073562 (+ 217us) server_negotiation.cc:407] Connection header received
0620 11:43:01.074099 (+ 537us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 11:43:01.074100 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 11:43:01.074110 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 11:43:01.074163 (+ 53us) server_negotiation.cc:226] Negotiated authn=SASL
0620 11:43:01.075043 (+ 880us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 11:43:01.075533 (+ 490us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 11:43:01.076536 (+ 1003us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 11:43:01.076742 (+ 206us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 11:43:01.076811 (+ 69us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 11:43:01.079075 (+ 2264us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 11:43:01.079077 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 11:43:01.079077 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 11:43:01.079085 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0620 11:43:01.080800 (+ 1715us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 11:43:01.081315 (+ 515us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 11:43:01.081316 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 11:43:01.081316 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 11:43:01.081374 (+ 58us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 11:43:01.081722 (+ 348us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 11:43:01.081723 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 11:43:01.081724 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 11:43:01.081847 (+ 123us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 11:43:01.081907 (+ 60us) server_negotiation.cc:1035] Waiting for connection context
0620 11:43:01.082303 (+ 396us) server_negotiation.cc:299] Negotiation successful
0620 11:43:01.082429 (+ 126us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":41}
I20260620 11:43:01.083091 15576 heartbeater.cc:347] Connected to a master server at 127.6.234.254:38223
I20260620 11:43:01.083189 15576 heartbeater.cc:464] Registering TS with master...
I20260620 11:43:01.083389 15576 heartbeater.cc:511] Master 127.6.234.254:38223 requested a full tablet report, sending...
I20260620 11:43:01.083771 15251 ts_manager.cc:194] Registered new tserver with Master: 98fae7d958fb4725b75f7b70e8b8d690 (127.6.234.194:45005)
I20260620 11:43:01.084162 15251 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.6.234.194@KRBTEST.COM'} at 127.6.234.194:50199
WARNING: no policy specified for kudu/127.6.234.195@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.6.234.195@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.6.234.195 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task1JRg7N/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781955746769051-7083-0/krb5kdc/kudu_127.6.234.195.keytab.
Entry for principal kudu/127.6.234.195 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task1JRg7N/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781955746769051-7083-0/krb5kdc/kudu_127.6.234.195.keytab.
I20260620 11:43:01.106386 7083 external_mini_cluster.cc:1326] Running /tmp/dist-test-task1JRg7N/build/release/bin/kudu
/tmp/dist-test-task1JRg7N/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task1JRg7N/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781955746769051-7083-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-task1JRg7N/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781955746769051-7083-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task1JRg7N/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781955746769051-7083-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-task1JRg7N/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781955746769051-7083-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.6.234.195:0
--local_ip_for_outbound_sockets=127.6.234.195
--webserver_interface=127.6.234.195
--webserver_port=0
--tserver_master_addrs=127.6.234.254:38223
--keytab_file=/tmp/dist-test-task1JRg7N/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781955746769051-7083-0/krb5kdc/kudu_127.6.234.195.keytab
--principal=kudu/127.6.234.195
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.6.234.212:40309
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task1JRg7N/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781955746769051-7083-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task1JRg7N/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781955746769051-7083-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task1JRg7N/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781955746769051-7083-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 11:43:01.189165 15581 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 11:43:01.189407 15581 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 11:43:01.189450 15581 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 11:43:01.190666 15581 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 11:43:01.190737 15581 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 11:43:01.190780 15581 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.6.234.195
I20260620 11:43:01.192198 15581 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.6.234.212:40309
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task1JRg7N/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781955746769051-7083-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-task1JRg7N/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781955746769051-7083-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task1JRg7N/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781955746769051-7083-0/krb5kdc/kudu_127.6.234.195.keytab
--principal=kudu/127.6.234.195
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.6.234.195:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task1JRg7N/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781955746769051-7083-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.6.234.195
--webserver_port=0
--tserver_master_addrs=127.6.234.254:38223
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.15581
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.6.234.195
--log_dir=/tmp/dist-test-task1JRg7N/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781955746769051-7083-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 20 Jun 2026 10:32:37 UTC on ccycloud.quasar-bgyofv.root.comops.site
build id 4457173
I20260620 11:43:01.192464 15581 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 11:43:01.192704 15581 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 11:43:01.197716 15586 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 11:43:01.197754 15587 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 11:43:01.197736 15581 server_base.cc:1034] running on GCE node
W20260620 11:43:01.197835 15589 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 11:43:01.198185 15581 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 11:43:01.198395 15581 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 11:43:01.199533 15581 hybrid_clock.cc:648] HybridClock initialized: now 1781955781199513 us; error 43 us; skew 500 ppm
Jun 20 11:43:01 dist-test-slave-cjc6 krb5kdc[15206](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781955781, etypes {rep=17 tkt=17 ses=17}, kudu/127.6.234.195@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 11:43:01.202318 15581 init.cc:377] Logged in from keytab as kudu/127.6.234.195@KRBTEST.COM (short username kudu)
I20260620 11:43:01.202765 15581 webserver.cc:460] Webserver started at http://127.6.234.195:36219/ using document root <none> and password file <none>
I20260620 11:43:01.203011 15581 fs_manager.cc:359] Metadata directory not provided
I20260620 11:43:01.203058 15581 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 11:43:01.203188 15581 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 11:43:01.204268 15581 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task1JRg7N/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781955746769051-7083-0/minicluster-data/ts-2/data/instance:
uuid: "4413292a043946a0bc50ce7d86758e85"
format_stamp: "Formatted at 2026-06-20 11:43:01 on dist-test-slave-cjc6"
server_key: "e23c6e40278074b05043da33872dd5e8"
server_key_iv: "883c5dfbfe1213412b0a0f781b0ace4d"
server_key_version: "encryptionkey@0"
I20260620 11:43:01.204710 15581 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task1JRg7N/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781955746769051-7083-0/minicluster-data/ts-2/wal/instance:
uuid: "4413292a043946a0bc50ce7d86758e85"
format_stamp: "Formatted at 2026-06-20 11:43:01 on dist-test-slave-cjc6"
server_key: "e23c6e40278074b05043da33872dd5e8"
server_key_iv: "883c5dfbfe1213412b0a0f781b0ace4d"
server_key_version: "encryptionkey@0"
I20260620 11:43:01.206063 15581 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260620 11:43:01.206862 15596 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 11:43:01.207056 15581 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260620 11:43:01.207120 15581 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task1JRg7N/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781955746769051-7083-0/minicluster-data/ts-2/data,/tmp/dist-test-task1JRg7N/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781955746769051-7083-0/minicluster-data/ts-2/wal
uuid: "4413292a043946a0bc50ce7d86758e85"
format_stamp: "Formatted at 2026-06-20 11:43:01 on dist-test-slave-cjc6"
server_key: "e23c6e40278074b05043da33872dd5e8"
server_key_iv: "883c5dfbfe1213412b0a0f781b0ace4d"
server_key_version: "encryptionkey@0"
I20260620 11:43:01.207214 15581 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task1JRg7N/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781955746769051-7083-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-task1JRg7N/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781955746769051-7083-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-task1JRg7N/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781955746769051-7083-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 11:43:01.213869 15581 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 11:43:01.216077 15581 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 11:43:01.216240 15581 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 11:43:01.216475 15581 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 11:43:01.216764 15581 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 11:43:01.216797 15581 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 11:43:01.216820 15581 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 11:43:01.216864 15581 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 11:43:01.222996 15581 rpc_server.cc:307] RPC server started. Bound to: 127.6.234.195:42495
I20260620 11:43:01.223088 15709 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.6.234.195:42495 every 8 connection(s)
I20260620 11:43:01.223506 15581 server_base.cc:1166] Dumped server information to /tmp/dist-test-task1JRg7N/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781955746769051-7083-0/minicluster-data/ts-2/data/info.pb
Jun 20 11:43:01 dist-test-slave-cjc6 krb5kdc[15206](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781955781, etypes {rep=17 tkt=17 ses=17}, kudu/127.6.234.195@KRBTEST.COM for kudu/127.6.234.254@KRBTEST.COM
I20260620 11:43:01.231207 7083 external_mini_cluster.cc:1388] Started /tmp/dist-test-task1JRg7N/build/release/bin/kudu as pid 15581
I20260620 11:43:01.231379 7083 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task1JRg7N/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781955746769051-7083-0/minicluster-data/ts-2/wal/instance
I20260620 11:43:01.231585 7083 external_mini_cluster.cc:1428] Setting key c816446a0daa5e9a7a69f019ad07ffc2
I20260620 11:43:01.233014 15290 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 11:43:01.224211 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.6.234.195:34581 (local address 127.6.234.254:38223)
0620 11:43:01.224287 (+ 76us) server_negotiation.cc:206] Beginning negotiation
0620 11:43:01.224289 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 11:43:01.224444 (+ 155us) server_negotiation.cc:407] Connection header received
0620 11:43:01.224939 (+ 495us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 11:43:01.224940 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 11:43:01.224951 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 11:43:01.225016 (+ 65us) server_negotiation.cc:226] Negotiated authn=SASL
0620 11:43:01.226000 (+ 984us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 11:43:01.226516 (+ 516us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 11:43:01.227540 (+ 1024us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 11:43:01.227723 (+ 183us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 11:43:01.227779 (+ 56us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 11:43:01.230086 (+ 2307us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 11:43:01.230088 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 11:43:01.230088 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 11:43:01.230095 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0620 11:43:01.231595 (+ 1500us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 11:43:01.232053 (+ 458us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 11:43:01.232055 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 11:43:01.232055 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 11:43:01.232114 (+ 59us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 11:43:01.232380 (+ 266us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 11:43:01.232381 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 11:43:01.232381 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 11:43:01.232503 (+ 122us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 11:43:01.232579 (+ 76us) server_negotiation.cc:1035] Waiting for connection context
0620 11:43:01.232844 (+ 265us) server_negotiation.cc:299] Negotiation successful
0620 11:43:01.232921 (+ 77us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":46}
I20260620 11:43:01.233222 15712 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 11:43:01.224167 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.234.254:38223 (local address 127.6.234.195:34581)
0620 11:43:01.224326 (+ 159us) negotiation.cc:107] Waiting for socket to connect
0620 11:43:01.224361 (+ 35us) client_negotiation.cc:174] Beginning negotiation
0620 11:43:01.224825 (+ 464us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 11:43:01.225071 (+ 246us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 11:43:01.225074 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 11:43:01.225630 (+ 556us) client_negotiation.cc:189] Negotiated authn=SASL
0620 11:43:01.225909 (+ 279us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 11:43:01.225914 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 11:43:01.226626 (+ 712us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 11:43:01.226627 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 11:43:01.227456 (+ 829us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 11:43:01.227458 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 11:43:01.227826 (+ 368us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 11:43:01.227827 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 11:43:01.227945 (+ 118us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 11:43:01.228524 (+ 579us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 11:43:01.228532 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0620 11:43:01.229998 (+ 1466us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 11:43:01.231723 (+ 1725us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 11:43:01.231731 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 11:43:01.231733 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0620 11:43:01.231965 (+ 232us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 11:43:01.232198 (+ 233us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 11:43:01.232199 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 11:43:01.232199 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0620 11:43:01.232304 (+ 105us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 11:43:01.232569 (+ 265us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 11:43:01.232575 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 11:43:01.232713 (+ 138us) client_negotiation.cc:769] Sending connection context
0620 11:43:01.232822 (+ 109us) client_negotiation.cc:240] Negotiation successful
0620 11:43:01.232982 (+ 160us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":123,"thread_start_us":97,"threads_started":1}
I20260620 11:43:01.233737 15710 heartbeater.cc:347] Connected to a master server at 127.6.234.254:38223
I20260620 11:43:01.233842 15710 heartbeater.cc:464] Registering TS with master...
I20260620 11:43:01.233985 15710 heartbeater.cc:511] Master 127.6.234.254:38223 requested a full tablet report, sending...
I20260620 11:43:01.234341 15251 ts_manager.cc:194] Registered new tserver with Master: 4413292a043946a0bc50ce7d86758e85 (127.6.234.195:42495)
I20260620 11:43:01.234673 15251 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.6.234.195@KRBTEST.COM'} at 127.6.234.195:34581
I20260620 11:43:01.243357 7083 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
WARNING: no policy specified for oryx/127.6.234.193@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.6.234.193@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.6.234.193 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task1JRg7N/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781955746769051-7083-0/krb5kdc/oryx_127.6.234.193.keytab.
Entry for principal oryx/127.6.234.193 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task1JRg7N/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781955746769051-7083-0/krb5kdc/oryx_127.6.234.193.keytab.
WARNING: no policy specified for oryx/127.6.234.194@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.6.234.194@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.6.234.194 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task1JRg7N/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781955746769051-7083-0/krb5kdc/oryx_127.6.234.194.keytab.
Entry for principal oryx/127.6.234.194 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task1JRg7N/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781955746769051-7083-0/krb5kdc/oryx_127.6.234.194.keytab.
WARNING: no policy specified for oryx/127.6.234.195@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.6.234.195@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.6.234.195 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task1JRg7N/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781955746769051-7083-0/krb5kdc/oryx_127.6.234.195.keytab.
Entry for principal oryx/127.6.234.195 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task1JRg7N/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781955746769051-7083-0/krb5kdc/oryx_127.6.234.195.keytab.
I20260620 11:43:01.312292 7083 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task1JRg7N/build/release/bin/kudu with pid 15313
I20260620 11:43:01.317013 7083 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task1JRg7N/build/release/bin/kudu with pid 15447
I20260620 11:43:01.320982 7083 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task1JRg7N/build/release/bin/kudu with pid 15581
I20260620 11:43:01.325337 7083 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task1JRg7N/build/release/bin/kudu with pid 15219
I20260620 11:43:01.330538 7083 external_mini_cluster.cc:1326] Running /tmp/dist-test-task1JRg7N/build/release/bin/kudu
/tmp/dist-test-task1JRg7N/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task1JRg7N/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781955746769051-7083-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task1JRg7N/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781955746769051-7083-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task1JRg7N/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781955746769051-7083-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task1JRg7N/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781955746769051-7083-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.6.234.254:38223
--webserver_interface=127.6.234.254
--webserver_port=41159
--keytab_file=/tmp/dist-test-task1JRg7N/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781955746769051-7083-0/krb5kdc/kudu_127.6.234.254.keytab
--principal=kudu/127.6.234.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.6.234.212:40309
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.6.234.254:38223
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task1JRg7N/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781955746769051-7083-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task1JRg7N/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781955746769051-7083-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task1JRg7N/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781955746769051-7083-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 11:43:01.410902 15719 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 11:43:01.411110 15719 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 11:43:01.411134 15719 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 11:43:01.412340 15719 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 11:43:01.412386 15719 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 11:43:01.412401 15719 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 11:43:01.412413 15719 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 11:43:01.412425 15719 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 11:43:01.413957 15719 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.6.234.212:40309
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task1JRg7N/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781955746769051-7083-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task1JRg7N/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781955746769051-7083-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.6.234.254:38223
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task1JRg7N/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781955746769051-7083-0/krb5kdc/kudu_127.6.234.254.keytab
--principal=kudu/127.6.234.254
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.6.234.254:38223
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task1JRg7N/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781955746769051-7083-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.6.234.254
--webserver_port=41159
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.15719
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task1JRg7N/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781955746769051-7083-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 20 Jun 2026 10:32:37 UTC on ccycloud.quasar-bgyofv.root.comops.site
build id 4457173
I20260620 11:43:01.414149 15719 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 11:43:01.414326 15719 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 11:43:01.419078 15727 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 11:43:01.419092 15725 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 11:43:01.419250 15719 server_base.cc:1034] running on GCE node
W20260620 11:43:01.419092 15724 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 11:43:01.419682 15719 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 11:43:01.419929 15719 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 11:43:01.421109 15719 hybrid_clock.cc:648] HybridClock initialized: now 1781955781421090 us; error 29 us; skew 500 ppm
Jun 20 11:43:01 dist-test-slave-cjc6 krb5kdc[15206](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781955781, etypes {rep=17 tkt=17 ses=17}, kudu/127.6.234.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 11:43:01.423818 15719 init.cc:377] Logged in from keytab as kudu/127.6.234.254@KRBTEST.COM (short username kudu)
I20260620 11:43:01.424288 15719 webserver.cc:460] Webserver started at http://127.6.234.254:41159/ using document root <none> and password file <none>
I20260620 11:43:01.424547 15719 fs_manager.cc:359] Metadata directory not provided
I20260620 11:43:01.424607 15719 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 11:43:01.426011 15719 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.000s sys 0.002s
I20260620 11:43:01.426718 15734 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 11:43:01.426916 15719 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.001s
I20260620 11:43:01.427008 15719 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task1JRg7N/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781955746769051-7083-0/minicluster-data/master-0/data,/tmp/dist-test-task1JRg7N/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781955746769051-7083-0/minicluster-data/master-0/wal
uuid: "9505661ea1f54dcbb8c78592e5491859"
format_stamp: "Formatted at 2026-06-20 11:43:00 on dist-test-slave-cjc6"
server_key: "27b6cc1fead2a7a7b64f721e86190630"
server_key_iv: "ea3e840295d2223970c4cff32110f6f0"
server_key_version: "encryptionkey@0"
I20260620 11:43:01.427371 15719 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task1JRg7N/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781955746769051-7083-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task1JRg7N/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781955746769051-7083-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task1JRg7N/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781955746769051-7083-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 11:43:01.442713 15719 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 11:43:01.444885 15719 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 11:43:01.445012 15719 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 11:43:01.449083 15719 rpc_server.cc:307] RPC server started. Bound to: 127.6.234.254:38223
I20260620 11:43:01.449115 15786 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.6.234.254:38223 every 8 connection(s)
I20260620 11:43:01.449509 15719 server_base.cc:1166] Dumped server information to /tmp/dist-test-task1JRg7N/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781955746769051-7083-0/minicluster-data/master-0/data/info.pb
I20260620 11:43:01.451092 15787 sys_catalog.cc:263] Verifying existing consensus state
I20260620 11:43:01.451794 15787 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 9505661ea1f54dcbb8c78592e5491859: Bootstrap starting.
I20260620 11:43:01.455312 7083 external_mini_cluster.cc:1388] Started /tmp/dist-test-task1JRg7N/build/release/bin/kudu as pid 15719
I20260620 11:43:01.455694 7083 external_mini_cluster.cc:1326] Running /tmp/dist-test-task1JRg7N/build/release/bin/kudu
/tmp/dist-test-task1JRg7N/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task1JRg7N/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781955746769051-7083-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-task1JRg7N/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781955746769051-7083-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task1JRg7N/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781955746769051-7083-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-task1JRg7N/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781955746769051-7083-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.6.234.193:38929
--local_ip_for_outbound_sockets=127.6.234.193
--tserver_master_addrs=127.6.234.254:38223
--webserver_port=42849
--webserver_interface=127.6.234.193
--keytab_file=/tmp/dist-test-task1JRg7N/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781955746769051-7083-0/krb5kdc/kudu_127.6.234.193.keytab
--principal=kudu/127.6.234.193
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.6.234.212:40309
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--principal=oryx/127.6.234.193
--keytab_file=/tmp/dist-test-task1JRg7N/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781955746769051-7083-0/krb5kdc/oryx_127.6.234.193.keytab with env {KRB5CCNAME=/tmp/dist-test-task1JRg7N/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781955746769051-7083-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task1JRg7N/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781955746769051-7083-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task1JRg7N/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781955746769051-7083-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260620 11:43:01.460634 15787 log.cc:824] T 00000000000000000000000000000000 P 9505661ea1f54dcbb8c78592e5491859: Log is configured to *not* fsync() on all Append() calls
I20260620 11:43:01.462477 15787 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 9505661ea1f54dcbb8c78592e5491859: Bootstrap replayed 1/1 log segments. Stats: ops{read=4 overwritten=0 applied=4 ignored=0} inserts{seen=3 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260620 11:43:01.462786 15787 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 9505661ea1f54dcbb8c78592e5491859: Bootstrap complete.
I20260620 11:43:01.465114 15787 raft_consensus.cc:348] T 00000000000000000000000000000000 P 9505661ea1f54dcbb8c78592e5491859 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9505661ea1f54dcbb8c78592e5491859" member_type: VOTER last_known_addr { host: "127.6.234.254" port: 38223 } }
I20260620 11:43:01.465348 15787 raft_consensus.cc:729] T 00000000000000000000000000000000 P 9505661ea1f54dcbb8c78592e5491859 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9505661ea1f54dcbb8c78592e5491859, State: Initialized, Role: FOLLOWER
I20260620 11:43:01.465608 15787 consensus_queue.cc:260] T 00000000000000000000000000000000 P 9505661ea1f54dcbb8c78592e5491859 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9505661ea1f54dcbb8c78592e5491859" member_type: VOTER last_known_addr { host: "127.6.234.254" port: 38223 } }
I20260620 11:43:01.465730 15787 raft_consensus.cc:388] T 00000000000000000000000000000000 P 9505661ea1f54dcbb8c78592e5491859 [term 1 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260620 11:43:01.465792 15787 raft_consensus.cc:482] T 00000000000000000000000000000000 P 9505661ea1f54dcbb8c78592e5491859 [term 1 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260620 11:43:01.465838 15787 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 9505661ea1f54dcbb8c78592e5491859 [term 1 FOLLOWER]: Advancing to term 2
I20260620 11:43:01.466895 15787 raft_consensus.cc:504] T 00000000000000000000000000000000 P 9505661ea1f54dcbb8c78592e5491859 [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9505661ea1f54dcbb8c78592e5491859" member_type: VOTER last_known_addr { host: "127.6.234.254" port: 38223 } }
I20260620 11:43:01.466984 15787 leader_election.cc:302] T 00000000000000000000000000000000 P 9505661ea1f54dcbb8c78592e5491859 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 9505661ea1f54dcbb8c78592e5491859; no voters:
I20260620 11:43:01.467204 15787 leader_election.cc:288] T 00000000000000000000000000000000 P 9505661ea1f54dcbb8c78592e5491859 [CANDIDATE]: Term 2 election: Requested vote from peers
I20260620 11:43:01.467244 15791 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 9505661ea1f54dcbb8c78592e5491859 [term 2 FOLLOWER]: Leader election won for term 2
I20260620 11:43:01.467365 15791 raft_consensus.cc:686] T 00000000000000000000000000000000 P 9505661ea1f54dcbb8c78592e5491859 [term 2 LEADER]: Becoming Leader. State: Replica: 9505661ea1f54dcbb8c78592e5491859, State: Running, Role: LEADER
I20260620 11:43:01.467448 15791 consensus_queue.cc:237] T 00000000000000000000000000000000 P 9505661ea1f54dcbb8c78592e5491859 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 2, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9505661ea1f54dcbb8c78592e5491859" member_type: VOTER last_known_addr { host: "127.6.234.254" port: 38223 } }
I20260620 11:43:01.467486 15787 sys_catalog.cc:564] T 00000000000000000000000000000000 P 9505661ea1f54dcbb8c78592e5491859 [sys.catalog]: configured and running, proceeding with master startup.
I20260620 11:43:01.467710 15791 sys_catalog.cc:455] T 00000000000000000000000000000000 P 9505661ea1f54dcbb8c78592e5491859 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 2 leader_uuid: "9505661ea1f54dcbb8c78592e5491859" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9505661ea1f54dcbb8c78592e5491859" member_type: VOTER last_known_addr { host: "127.6.234.254" port: 38223 } } }
I20260620 11:43:01.467778 15791 sys_catalog.cc:458] T 00000000000000000000000000000000 P 9505661ea1f54dcbb8c78592e5491859 [sys.catalog]: This master's current role is: LEADER
I20260620 11:43:01.467890 15791 sys_catalog.cc:455] T 00000000000000000000000000000000 P 9505661ea1f54dcbb8c78592e5491859 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 9505661ea1f54dcbb8c78592e5491859. Latest consensus state: current_term: 2 leader_uuid: "9505661ea1f54dcbb8c78592e5491859" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9505661ea1f54dcbb8c78592e5491859" member_type: VOTER last_known_addr { host: "127.6.234.254" port: 38223 } } }
I20260620 11:43:01.467958 15791 sys_catalog.cc:458] T 00000000000000000000000000000000 P 9505661ea1f54dcbb8c78592e5491859 [sys.catalog]: This master's current role is: LEADER
I20260620 11:43:01.468410 15795 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260620 11:43:01.468940 15795 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260620 11:43:01.469086 15795 catalog_manager.cc:1259] Loaded cluster ID: 74e1ace3749840ab977b124557aa3e82
I20260620 11:43:01.469129 15795 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260620 11:43:01.470125 15795 catalog_manager.cc:1504] Loading token signing keys...
I20260620 11:43:01.470316 15795 catalog_manager.cc:5879] T 00000000000000000000000000000000 P 9505661ea1f54dcbb8c78592e5491859: Loaded TSK: 0
I20260620 11:43:01.470526 15795 catalog_manager.cc:1514] Initializing in-progress tserver states...
W20260620 11:43:01.550236 15789 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 11:43:01.550451 15789 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 11:43:01.550475 15789 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 11:43:01.551692 15789 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 11:43:01.551741 15789 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 11:43:01.551769 15789 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.6.234.193
I20260620 11:43:01.553224 15789 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.6.234.212:40309
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task1JRg7N/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781955746769051-7083-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-task1JRg7N/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781955746769051-7083-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task1JRg7N/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781955746769051-7083-0/krb5kdc/oryx_127.6.234.193.keytab
--principal=oryx/127.6.234.193
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.6.234.193:38929
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task1JRg7N/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781955746769051-7083-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.6.234.193
--webserver_port=42849
--tserver_master_addrs=127.6.234.254:38223
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.15789
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.6.234.193
--log_dir=/tmp/dist-test-task1JRg7N/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781955746769051-7083-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 20 Jun 2026 10:32:37 UTC on ccycloud.quasar-bgyofv.root.comops.site
build id 4457173
I20260620 11:43:01.553464 15789 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 11:43:01.553784 15789 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 11:43:01.558672 15813 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 11:43:01.558656 15816 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 11:43:01.558645 15814 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 11:43:01.559060 15789 server_base.cc:1034] running on GCE node
I20260620 11:43:01.559280 15789 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 11:43:01.559537 15789 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 11:43:01.560741 15789 hybrid_clock.cc:648] HybridClock initialized: now 1781955781560715 us; error 41 us; skew 500 ppm
Jun 20 11:43:01 dist-test-slave-cjc6 krb5kdc[15206](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781955781, etypes {rep=17 tkt=17 ses=17}, oryx/127.6.234.193@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 11:43:01.563351 15789 init.cc:377] Logged in from keytab as oryx/127.6.234.193@KRBTEST.COM (short username oryx)
I20260620 11:43:01.563874 15789 webserver.cc:460] Webserver started at http://127.6.234.193:42849/ using document root <none> and password file <none>
I20260620 11:43:01.564122 15789 fs_manager.cc:359] Metadata directory not provided
I20260620 11:43:01.564194 15789 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 11:43:01.565572 15789 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.001s sys 0.000s
I20260620 11:43:01.566277 15823 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 11:43:01.566449 15789 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.000s
I20260620 11:43:01.566534 15789 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task1JRg7N/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781955746769051-7083-0/minicluster-data/ts-0/data,/tmp/dist-test-task1JRg7N/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781955746769051-7083-0/minicluster-data/ts-0/wal
uuid: "6e169ae0a436470fb4a87f555360e93e"
format_stamp: "Formatted at 2026-06-20 11:43:00 on dist-test-slave-cjc6"
server_key: "237554ec3b69d3aa0f49551f5b435622"
server_key_iv: "05cdb51a5d34be71ae7e73d2948be6a2"
server_key_version: "encryptionkey@0"
I20260620 11:43:01.566879 15789 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task1JRg7N/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781955746769051-7083-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-task1JRg7N/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781955746769051-7083-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-task1JRg7N/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781955746769051-7083-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 11:43:01.595541 15789 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 11:43:01.598186 15789 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 11:43:01.598340 15789 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 11:43:01.598635 15789 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 11:43:01.598997 15789 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 11:43:01.599043 15789 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 11:43:01.599066 15789 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 11:43:01.599109 15789 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 11:43:01.606253 15789 rpc_server.cc:307] RPC server started. Bound to: 127.6.234.193:38929
I20260620 11:43:01.606302 15936 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.6.234.193:38929 every 8 connection(s)
I20260620 11:43:01.606666 15789 server_base.cc:1166] Dumped server information to /tmp/dist-test-task1JRg7N/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781955746769051-7083-0/minicluster-data/ts-0/data/info.pb
I20260620 11:43:01.611522 7083 external_mini_cluster.cc:1388] Started /tmp/dist-test-task1JRg7N/build/release/bin/kudu as pid 15789
I20260620 11:43:01.611888 7083 external_mini_cluster.cc:1326] Running /tmp/dist-test-task1JRg7N/build/release/bin/kudu
/tmp/dist-test-task1JRg7N/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task1JRg7N/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781955746769051-7083-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-task1JRg7N/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781955746769051-7083-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task1JRg7N/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781955746769051-7083-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-task1JRg7N/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781955746769051-7083-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.6.234.194:45005
--local_ip_for_outbound_sockets=127.6.234.194
--tserver_master_addrs=127.6.234.254:38223
--webserver_port=40853
--webserver_interface=127.6.234.194
--keytab_file=/tmp/dist-test-task1JRg7N/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781955746769051-7083-0/krb5kdc/kudu_127.6.234.194.keytab
--principal=kudu/127.6.234.194
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.6.234.212:40309
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--principal=oryx/127.6.234.194
--keytab_file=/tmp/dist-test-task1JRg7N/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781955746769051-7083-0/krb5kdc/oryx_127.6.234.194.keytab with env {KRB5CCNAME=/tmp/dist-test-task1JRg7N/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781955746769051-7083-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task1JRg7N/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781955746769051-7083-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task1JRg7N/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781955746769051-7083-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Jun 20 11:43:01 dist-test-slave-cjc6 krb5kdc[15206](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.6.234.193@KRBTEST.COM for oryx/127.6.234.254@KRBTEST.COM, Server not found in Kerberos database
Jun 20 11:43:01 dist-test-slave-cjc6 krb5kdc[15206](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.6.234.193@KRBTEST.COM for oryx/127.6.234.254@KRBTEST.COM, Server not found in Kerberos database
I20260620 11:43:01.614758 15939 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 11:43:01.607334 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.234.254:38223 (local address 127.6.234.193:40555)
0620 11:43:01.607504 (+ 170us) negotiation.cc:107] Waiting for socket to connect
0620 11:43:01.607545 (+ 41us) client_negotiation.cc:174] Beginning negotiation
0620 11:43:01.608014 (+ 469us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 11:43:01.608396 (+ 382us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 11:43:01.608400 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 11:43:01.608958 (+ 558us) client_negotiation.cc:189] Negotiated authn=SASL
0620 11:43:01.609150 (+ 192us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 11:43:01.609155 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 11:43:01.610269 (+ 1114us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 11:43:01.610270 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 11:43:01.611034 (+ 764us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 11:43:01.611035 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 11:43:01.611737 (+ 702us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 11:43:01.611737 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 11:43:01.611844 (+ 107us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 11:43:01.612487 (+ 643us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 11:43:01.612495 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0620 11:43:01.614617 (+ 2122us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.6.234.254:38223: Server oryx/127.6.234.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":131,"thread_start_us":97,"threads_started":1}
W20260620 11:43:01.614976 15940 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 11:43:01.607385 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.6.234.193:40555 (local address 127.6.234.254:38223)
0620 11:43:01.607552 (+ 167us) server_negotiation.cc:206] Beginning negotiation
0620 11:43:01.607561 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0620 11:43:01.607624 (+ 63us) server_negotiation.cc:407] Connection header received
0620 11:43:01.608170 (+ 546us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 11:43:01.608175 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 11:43:01.608224 (+ 49us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 11:43:01.608345 (+ 121us) server_negotiation.cc:226] Negotiated authn=SASL
0620 11:43:01.609252 (+ 907us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 11:43:01.610180 (+ 928us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 11:43:01.611107 (+ 927us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 11:43:01.611638 (+ 531us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 11:43:01.611752 (+ 114us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 11:43:01.614780 (+ 3028us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.6.234.193:40555: BlockingRecv error: recv got EOF from 127.6.234.193:40555 (error 108)
Metrics: {"server-negotiator.queue_time_us":121,"thread_start_us":88,"threads_started":1}
W20260620 11:43:01.615293 15937 heartbeater.cc:644] Failed to heartbeat to 127.6.234.254:38223 (0 consecutive failures): Not authorized: Failed to ping master at 127.6.234.254:38223: Client connection negotiation failed: client connection to 127.6.234.254:38223: Server oryx/127.6.234.254@KRBTEST.COM not found in Kerberos database
Jun 20 11:43:01 dist-test-slave-cjc6 krb5kdc[15206](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.6.234.193@KRBTEST.COM for oryx/127.6.234.254@KRBTEST.COM, Server not found in Kerberos database
Jun 20 11:43:01 dist-test-slave-cjc6 krb5kdc[15206](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.6.234.193@KRBTEST.COM for oryx/127.6.234.254@KRBTEST.COM, Server not found in Kerberos database
I20260620 11:43:01.619946 15939 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 11:43:01.615608 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.234.254:38223 (local address 127.6.234.193:55131)
0620 11:43:01.615658 (+ 50us) negotiation.cc:107] Waiting for socket to connect
0620 11:43:01.615669 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0620 11:43:01.615710 (+ 41us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 11:43:01.615866 (+ 156us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 11:43:01.615867 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 11:43:01.615995 (+ 128us) client_negotiation.cc:189] Negotiated authn=SASL
0620 11:43:01.616035 (+ 40us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 11:43:01.616036 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 11:43:01.616651 (+ 615us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 11:43:01.616652 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 11:43:01.617152 (+ 500us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 11:43:01.617153 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 11:43:01.617511 (+ 358us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 11:43:01.617512 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 11:43:01.617577 (+ 65us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 11:43:01.618074 (+ 497us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 11:43:01.618078 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0620 11:43:01.619870 (+ 1792us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.6.234.254:38223: Server oryx/127.6.234.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":33}
W20260620 11:43:01.620002 15940 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 11:43:01.615667 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.6.234.193:55131 (local address 127.6.234.254:38223)
0620 11:43:01.615710 (+ 43us) server_negotiation.cc:206] Beginning negotiation
0620 11:43:01.615712 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 11:43:01.615721 (+ 9us) server_negotiation.cc:407] Connection header received
0620 11:43:01.615781 (+ 60us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 11:43:01.615782 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 11:43:01.615795 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 11:43:01.615865 (+ 70us) server_negotiation.cc:226] Negotiated authn=SASL
0620 11:43:01.616118 (+ 253us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 11:43:01.616575 (+ 457us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 11:43:01.617247 (+ 672us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 11:43:01.617439 (+ 192us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 11:43:01.617514 (+ 75us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 11:43:01.619940 (+ 2426us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.6.234.193:55131: BlockingRecv error: recv got EOF from 127.6.234.193:55131 (error 108)
Metrics: {"server-negotiator.queue_time_us":22}
Jun 20 11:43:01 dist-test-slave-cjc6 krb5kdc[15206](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.6.234.193@KRBTEST.COM for oryx/127.6.234.254@KRBTEST.COM, Server not found in Kerberos database
Jun 20 11:43:01 dist-test-slave-cjc6 krb5kdc[15206](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.6.234.193@KRBTEST.COM for oryx/127.6.234.254@KRBTEST.COM, Server not found in Kerberos database
I20260620 11:43:01.624631 15939 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 11:43:01.620259 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.234.254:38223 (local address 127.6.234.193:53345)
0620 11:43:01.620296 (+ 37us) negotiation.cc:107] Waiting for socket to connect
0620 11:43:01.620305 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0620 11:43:01.620346 (+ 41us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 11:43:01.620487 (+ 141us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 11:43:01.620487 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 11:43:01.620585 (+ 98us) client_negotiation.cc:189] Negotiated authn=SASL
0620 11:43:01.620608 (+ 23us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 11:43:01.620609 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 11:43:01.621522 (+ 913us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 11:43:01.621522 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 11:43:01.622036 (+ 514us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 11:43:01.622037 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 11:43:01.622489 (+ 452us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 11:43:01.622489 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 11:43:01.622524 (+ 35us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 11:43:01.622911 (+ 387us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 11:43:01.622914 (+ 3us) client_negotiation.cc:656] calling sasl_client_start()
0620 11:43:01.624566 (+ 1652us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.6.234.254:38223: Server oryx/127.6.234.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":24}
W20260620 11:43:01.624755 15940 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 11:43:01.620297 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.6.234.193:53345 (local address 127.6.234.254:38223)
0620 11:43:01.620365 (+ 68us) server_negotiation.cc:206] Beginning negotiation
0620 11:43:01.620367 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 11:43:01.620377 (+ 10us) server_negotiation.cc:407] Connection header received
0620 11:43:01.620399 (+ 22us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 11:43:01.620400 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 11:43:01.620410 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 11:43:01.620479 (+ 69us) server_negotiation.cc:226] Negotiated authn=SASL
0620 11:43:01.620678 (+ 199us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 11:43:01.621444 (+ 766us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 11:43:01.622109 (+ 665us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 11:43:01.622417 (+ 308us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 11:43:01.622486 (+ 69us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 11:43:01.624691 (+ 2205us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.6.234.193:53345: BlockingRecv error: recv got EOF from 127.6.234.193:53345 (error 108)
Metrics: {"server-negotiator.queue_time_us":42}
W20260620 11:43:01.624771 15937 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
W20260620 11:43:01.694129 15941 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 11:43:01.694331 15941 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 11:43:01.694355 15941 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 11:43:01.695549 15941 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 11:43:01.695621 15941 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 11:43:01.695667 15941 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.6.234.194
I20260620 11:43:01.698072 15941 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.6.234.212:40309
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task1JRg7N/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781955746769051-7083-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-task1JRg7N/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781955746769051-7083-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task1JRg7N/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781955746769051-7083-0/krb5kdc/oryx_127.6.234.194.keytab
--principal=oryx/127.6.234.194
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.6.234.194:45005
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task1JRg7N/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781955746769051-7083-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.6.234.194
--webserver_port=40853
--tserver_master_addrs=127.6.234.254:38223
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.15941
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.6.234.194
--log_dir=/tmp/dist-test-task1JRg7N/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781955746769051-7083-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 20 Jun 2026 10:32:37 UTC on ccycloud.quasar-bgyofv.root.comops.site
build id 4457173
I20260620 11:43:01.698349 15941 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 11:43:01.698645 15941 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 11:43:01.705778 15946 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 11:43:01.705752 15941 server_base.cc:1034] running on GCE node
W20260620 11:43:01.705914 15947 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 11:43:01.705623 15949 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 11:43:01.706491 15941 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 11:43:01.706796 15941 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 11:43:01.707967 15941 hybrid_clock.cc:648] HybridClock initialized: now 1781955781707954 us; error 30 us; skew 500 ppm
Jun 20 11:43:01 dist-test-slave-cjc6 krb5kdc[15206](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781955781, etypes {rep=17 tkt=17 ses=17}, oryx/127.6.234.194@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 11:43:01.710700 15941 init.cc:377] Logged in from keytab as oryx/127.6.234.194@KRBTEST.COM (short username oryx)
I20260620 11:43:01.711161 15941 webserver.cc:460] Webserver started at http://127.6.234.194:40853/ using document root <none> and password file <none>
I20260620 11:43:01.711398 15941 fs_manager.cc:359] Metadata directory not provided
I20260620 11:43:01.711481 15941 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 11:43:01.712850 15941 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.001s sys 0.000s
I20260620 11:43:01.713420 15956 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 11:43:01.713600 15941 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260620 11:43:01.713699 15941 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task1JRg7N/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781955746769051-7083-0/minicluster-data/ts-1/data,/tmp/dist-test-task1JRg7N/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781955746769051-7083-0/minicluster-data/ts-1/wal
uuid: "98fae7d958fb4725b75f7b70e8b8d690"
format_stamp: "Formatted at 2026-06-20 11:43:01 on dist-test-slave-cjc6"
server_key: "8420df0ab3929d45dbdfbfb2cc8650b2"
server_key_iv: "02d03e53234502c7c66c3159c63cfbfe"
server_key_version: "encryptionkey@0"
I20260620 11:43:01.714110 15941 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task1JRg7N/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781955746769051-7083-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-task1JRg7N/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781955746769051-7083-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-task1JRg7N/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781955746769051-7083-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 11:43:01.724406 15941 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 11:43:01.726778 15941 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 11:43:01.726984 15941 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 11:43:01.727229 15941 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 11:43:01.727535 15941 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 11:43:01.727568 15941 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 11:43:01.727591 15941 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 11:43:01.727635 15941 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 11:43:01.733732 15941 rpc_server.cc:307] RPC server started. Bound to: 127.6.234.194:45005
I20260620 11:43:01.733767 16069 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.6.234.194:45005 every 8 connection(s)
I20260620 11:43:01.734104 15941 server_base.cc:1166] Dumped server information to /tmp/dist-test-task1JRg7N/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781955746769051-7083-0/minicluster-data/ts-1/data/info.pb
I20260620 11:43:01.737159 7083 external_mini_cluster.cc:1388] Started /tmp/dist-test-task1JRg7N/build/release/bin/kudu as pid 15941
I20260620 11:43:01.737466 7083 external_mini_cluster.cc:1326] Running /tmp/dist-test-task1JRg7N/build/release/bin/kudu
/tmp/dist-test-task1JRg7N/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task1JRg7N/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781955746769051-7083-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-task1JRg7N/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781955746769051-7083-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task1JRg7N/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781955746769051-7083-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-task1JRg7N/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781955746769051-7083-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.6.234.195:42495
--local_ip_for_outbound_sockets=127.6.234.195
--tserver_master_addrs=127.6.234.254:38223
--webserver_port=36219
--webserver_interface=127.6.234.195
--keytab_file=/tmp/dist-test-task1JRg7N/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781955746769051-7083-0/krb5kdc/kudu_127.6.234.195.keytab
--principal=kudu/127.6.234.195
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.6.234.212:40309
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--principal=oryx/127.6.234.195
--keytab_file=/tmp/dist-test-task1JRg7N/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781955746769051-7083-0/krb5kdc/oryx_127.6.234.195.keytab with env {KRB5CCNAME=/tmp/dist-test-task1JRg7N/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781955746769051-7083-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task1JRg7N/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781955746769051-7083-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task1JRg7N/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781955746769051-7083-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Jun 20 11:43:01 dist-test-slave-cjc6 krb5kdc[15206](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.6.234.194@KRBTEST.COM for oryx/127.6.234.254@KRBTEST.COM, Server not found in Kerberos database
Jun 20 11:43:01 dist-test-slave-cjc6 krb5kdc[15206](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.6.234.194@KRBTEST.COM for oryx/127.6.234.254@KRBTEST.COM, Server not found in Kerberos database
I20260620 11:43:01.742133 16072 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 11:43:01.734768 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.234.254:38223 (local address 127.6.234.194:57049)
0620 11:43:01.734955 (+ 187us) negotiation.cc:107] Waiting for socket to connect
0620 11:43:01.734990 (+ 35us) client_negotiation.cc:174] Beginning negotiation
0620 11:43:01.735462 (+ 472us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 11:43:01.735674 (+ 212us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 11:43:01.735677 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 11:43:01.736228 (+ 551us) client_negotiation.cc:189] Negotiated authn=SASL
0620 11:43:01.736421 (+ 193us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 11:43:01.736426 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 11:43:01.737068 (+ 642us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 11:43:01.737069 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 11:43:01.738459 (+ 1390us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 11:43:01.738462 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 11:43:01.738870 (+ 408us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 11:43:01.738881 (+ 11us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 11:43:01.739061 (+ 180us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 11:43:01.739586 (+ 525us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 11:43:01.739595 (+ 9us) client_negotiation.cc:656] calling sasl_client_start()
0620 11:43:01.741996 (+ 2401us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.6.234.254:38223: Server oryx/127.6.234.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":150,"thread_start_us":92,"threads_started":1}
W20260620 11:43:01.742204 15940 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 11:43:01.734765 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.6.234.194:57049 (local address 127.6.234.254:38223)
0620 11:43:01.734834 (+ 69us) server_negotiation.cc:206] Beginning negotiation
0620 11:43:01.734836 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 11:43:01.735070 (+ 234us) server_negotiation.cc:407] Connection header received
0620 11:43:01.735543 (+ 473us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 11:43:01.735544 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 11:43:01.735557 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 11:43:01.735609 (+ 52us) server_negotiation.cc:226] Negotiated authn=SASL
0620 11:43:01.736493 (+ 884us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 11:43:01.736970 (+ 477us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 11:43:01.738559 (+ 1589us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 11:43:01.738774 (+ 215us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 11:43:01.738845 (+ 71us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 11:43:01.742114 (+ 3269us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.6.234.194:57049: BlockingRecv error: recv got EOF from 127.6.234.194:57049 (error 108)
Metrics: {"server-negotiator.queue_time_us":40}
W20260620 11:43:01.742393 16070 heartbeater.cc:644] Failed to heartbeat to 127.6.234.254:38223 (0 consecutive failures): Not authorized: Failed to ping master at 127.6.234.254:38223: Client connection negotiation failed: client connection to 127.6.234.254:38223: Server oryx/127.6.234.254@KRBTEST.COM not found in Kerberos database
Jun 20 11:43:01 dist-test-slave-cjc6 krb5kdc[15206](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.6.234.194@KRBTEST.COM for oryx/127.6.234.254@KRBTEST.COM, Server not found in Kerberos database
Jun 20 11:43:01 dist-test-slave-cjc6 krb5kdc[15206](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.6.234.194@KRBTEST.COM for oryx/127.6.234.254@KRBTEST.COM, Server not found in Kerberos database
I20260620 11:43:01.747243 16072 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 11:43:01.742634 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.234.254:38223 (local address 127.6.234.194:54759)
0620 11:43:01.742732 (+ 98us) negotiation.cc:107] Waiting for socket to connect
0620 11:43:01.742742 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0620 11:43:01.742778 (+ 36us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 11:43:01.742906 (+ 128us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 11:43:01.742906 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 11:43:01.743011 (+ 105us) client_negotiation.cc:189] Negotiated authn=SASL
0620 11:43:01.743051 (+ 40us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 11:43:01.743052 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 11:43:01.743979 (+ 927us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 11:43:01.743980 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 11:43:01.744460 (+ 480us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 11:43:01.744461 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 11:43:01.744934 (+ 473us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 11:43:01.744935 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 11:43:01.744978 (+ 43us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 11:43:01.745416 (+ 438us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 11:43:01.745420 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0620 11:43:01.747201 (+ 1781us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.6.234.254:38223: Server oryx/127.6.234.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":83}
W20260620 11:43:01.747399 15940 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 11:43:01.742688 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.6.234.194:54759 (local address 127.6.234.254:38223)
0620 11:43:01.742756 (+ 68us) server_negotiation.cc:206] Beginning negotiation
0620 11:43:01.742758 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 11:43:01.742771 (+ 13us) server_negotiation.cc:407] Connection header received
0620 11:43:01.742825 (+ 54us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 11:43:01.742826 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 11:43:01.742837 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 11:43:01.742894 (+ 57us) server_negotiation.cc:226] Negotiated authn=SASL
0620 11:43:01.743115 (+ 221us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 11:43:01.743895 (+ 780us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 11:43:01.744533 (+ 638us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 11:43:01.744861 (+ 328us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 11:43:01.744938 (+ 77us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 11:43:01.747312 (+ 2374us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.6.234.194:54759: BlockingRecv error: recv got EOF from 127.6.234.194:54759 (error 108)
Metrics: {"server-negotiator.queue_time_us":40}
Jun 20 11:43:01 dist-test-slave-cjc6 krb5kdc[15206](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.6.234.194@KRBTEST.COM for oryx/127.6.234.254@KRBTEST.COM, Server not found in Kerberos database
Jun 20 11:43:01 dist-test-slave-cjc6 krb5kdc[15206](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.6.234.194@KRBTEST.COM for oryx/127.6.234.254@KRBTEST.COM, Server not found in Kerberos database
I20260620 11:43:01.751896 16072 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 11:43:01.747543 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.234.254:38223 (local address 127.6.234.194:48477)
0620 11:43:01.747630 (+ 87us) negotiation.cc:107] Waiting for socket to connect
0620 11:43:01.747638 (+ 8us) client_negotiation.cc:174] Beginning negotiation
0620 11:43:01.747667 (+ 29us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 11:43:01.747847 (+ 180us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 11:43:01.747847 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 11:43:01.747945 (+ 98us) client_negotiation.cc:189] Negotiated authn=SASL
0620 11:43:01.747969 (+ 24us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 11:43:01.747969 (+ 0us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 11:43:01.748881 (+ 912us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 11:43:01.748882 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 11:43:01.749330 (+ 448us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 11:43:01.749331 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 11:43:01.749779 (+ 448us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 11:43:01.749780 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 11:43:01.749821 (+ 41us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 11:43:01.750200 (+ 379us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 11:43:01.750204 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0620 11:43:01.751835 (+ 1631us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.6.234.254:38223: Server oryx/127.6.234.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":66}
W20260620 11:43:01.751989 15940 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 11:43:01.747690 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.6.234.194:48477 (local address 127.6.234.254:38223)
0620 11:43:01.747728 (+ 38us) server_negotiation.cc:206] Beginning negotiation
0620 11:43:01.747729 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0620 11:43:01.747739 (+ 10us) server_negotiation.cc:407] Connection header received
0620 11:43:01.747761 (+ 22us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 11:43:01.747761 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 11:43:01.747771 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 11:43:01.747839 (+ 68us) server_negotiation.cc:226] Negotiated authn=SASL
0620 11:43:01.748039 (+ 200us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 11:43:01.748803 (+ 764us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 11:43:01.749394 (+ 591us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 11:43:01.749683 (+ 289us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 11:43:01.749776 (+ 93us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 11:43:01.751911 (+ 2135us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.6.234.194:48477: BlockingRecv error: recv got EOF from 127.6.234.194:48477 (error 108)
Metrics: {"server-negotiator.queue_time_us":20}
W20260620 11:43:01.752168 16070 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
W20260620 11:43:01.821290 16073 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 11:43:01.821511 16073 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 11:43:01.821549 16073 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 11:43:01.822947 16073 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 11:43:01.823036 16073 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 11:43:01.823083 16073 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.6.234.195
I20260620 11:43:01.824646 16073 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.6.234.212:40309
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task1JRg7N/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781955746769051-7083-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-task1JRg7N/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781955746769051-7083-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task1JRg7N/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781955746769051-7083-0/krb5kdc/oryx_127.6.234.195.keytab
--principal=oryx/127.6.234.195
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.6.234.195:42495
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task1JRg7N/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781955746769051-7083-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.6.234.195
--webserver_port=36219
--tserver_master_addrs=127.6.234.254:38223
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.16073
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.6.234.195
--log_dir=/tmp/dist-test-task1JRg7N/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781955746769051-7083-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 20 Jun 2026 10:32:37 UTC on ccycloud.quasar-bgyofv.root.comops.site
build id 4457173
I20260620 11:43:01.824927 16073 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 11:43:01.825151 16073 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 11:43:01.830107 16081 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 11:43:01.830178 16079 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 11:43:01.830116 16078 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 11:43:01.830345 16073 server_base.cc:1034] running on GCE node
I20260620 11:43:01.830570 16073 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 11:43:01.830807 16073 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 11:43:01.831967 16073 hybrid_clock.cc:648] HybridClock initialized: now 1781955781831961 us; error 35 us; skew 500 ppm
Jun 20 11:43:01 dist-test-slave-cjc6 krb5kdc[15206](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781955781, etypes {rep=17 tkt=17 ses=17}, oryx/127.6.234.195@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 11:43:01.834409 16073 init.cc:377] Logged in from keytab as oryx/127.6.234.195@KRBTEST.COM (short username oryx)
I20260620 11:43:01.834897 16073 webserver.cc:460] Webserver started at http://127.6.234.195:36219/ using document root <none> and password file <none>
I20260620 11:43:01.835117 16073 fs_manager.cc:359] Metadata directory not provided
I20260620 11:43:01.835198 16073 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 11:43:01.836521 16073 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.001s sys 0.000s
I20260620 11:43:01.837062 16088 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 11:43:01.837226 16073 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260620 11:43:01.837328 16073 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task1JRg7N/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781955746769051-7083-0/minicluster-data/ts-2/data,/tmp/dist-test-task1JRg7N/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781955746769051-7083-0/minicluster-data/ts-2/wal
uuid: "4413292a043946a0bc50ce7d86758e85"
format_stamp: "Formatted at 2026-06-20 11:43:01 on dist-test-slave-cjc6"
server_key: "e23c6e40278074b05043da33872dd5e8"
server_key_iv: "883c5dfbfe1213412b0a0f781b0ace4d"
server_key_version: "encryptionkey@0"
I20260620 11:43:01.837666 16073 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task1JRg7N/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781955746769051-7083-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-task1JRg7N/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781955746769051-7083-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-task1JRg7N/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781955746769051-7083-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 11:43:01.846627 16073 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 11:43:01.848408 16073 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 11:43:01.848560 16073 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 11:43:01.848778 16073 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 11:43:01.849064 16073 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 11:43:01.849117 16073 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 11:43:01.849156 16073 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 11:43:01.849184 16073 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 11:43:01.854962 16073 rpc_server.cc:307] RPC server started. Bound to: 127.6.234.195:42495
I20260620 11:43:01.855011 16201 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.6.234.195:42495 every 8 connection(s)
I20260620 11:43:01.855330 16073 server_base.cc:1166] Dumped server information to /tmp/dist-test-task1JRg7N/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781955746769051-7083-0/minicluster-data/ts-2/data/info.pb
Jun 20 11:43:01 dist-test-slave-cjc6 krb5kdc[15206](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.6.234.195@KRBTEST.COM for oryx/127.6.234.254@KRBTEST.COM, Server not found in Kerberos database
Jun 20 11:43:01 dist-test-slave-cjc6 krb5kdc[15206](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.6.234.195@KRBTEST.COM for oryx/127.6.234.254@KRBTEST.COM, Server not found in Kerberos database
I20260620 11:43:01.862519 7083 external_mini_cluster.cc:1388] Started /tmp/dist-test-task1JRg7N/build/release/bin/kudu as pid 16073
I20260620 11:43:01.862743 16204 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 11:43:01.855950 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.234.254:38223 (local address 127.6.234.195:34939)
0620 11:43:01.856077 (+ 127us) negotiation.cc:107] Waiting for socket to connect
0620 11:43:01.856125 (+ 48us) client_negotiation.cc:174] Beginning negotiation
0620 11:43:01.856619 (+ 494us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 11:43:01.856884 (+ 265us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 11:43:01.856888 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 11:43:01.857543 (+ 655us) client_negotiation.cc:189] Negotiated authn=SASL
0620 11:43:01.857794 (+ 251us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 11:43:01.857800 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 11:43:01.858484 (+ 684us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 11:43:01.858485 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 11:43:01.859292 (+ 807us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 11:43:01.859294 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 11:43:01.859678 (+ 384us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 11:43:01.859679 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 11:43:01.859809 (+ 130us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 11:43:01.860250 (+ 441us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 11:43:01.860258 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0620 11:43:01.862527 (+ 2269us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.6.234.254:38223: Server oryx/127.6.234.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":91,"thread_start_us":60,"threads_started":1}
W20260620 11:43:01.862741 15940 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 11:43:01.855989 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.6.234.195:34939 (local address 127.6.234.254:38223)
0620 11:43:01.856032 (+ 43us) server_negotiation.cc:206] Beginning negotiation
0620 11:43:01.856035 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0620 11:43:01.856204 (+ 169us) server_negotiation.cc:407] Connection header received
0620 11:43:01.856724 (+ 520us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 11:43:01.856726 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 11:43:01.856740 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 11:43:01.856812 (+ 72us) server_negotiation.cc:226] Negotiated authn=SASL
0620 11:43:01.857890 (+ 1078us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 11:43:01.858390 (+ 500us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 11:43:01.859395 (+ 1005us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 11:43:01.859599 (+ 204us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 11:43:01.859673 (+ 74us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 11:43:01.862665 (+ 2992us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.6.234.195:34939: BlockingRecv error: recv got EOF from 127.6.234.195:34939 (error 108)
Metrics: {"server-negotiator.queue_time_us":17}
W20260620 11:43:01.862990 16202 heartbeater.cc:644] Failed to heartbeat to 127.6.234.254:38223 (0 consecutive failures): Not authorized: Failed to ping master at 127.6.234.254:38223: Client connection negotiation failed: client connection to 127.6.234.254:38223: Server oryx/127.6.234.254@KRBTEST.COM not found in Kerberos database
Jun 20 11:43:01 dist-test-slave-cjc6 krb5kdc[15206](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.6.234.195@KRBTEST.COM for oryx/127.6.234.254@KRBTEST.COM, Server not found in Kerberos database
Jun 20 11:43:01 dist-test-slave-cjc6 krb5kdc[15206](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.6.234.195@KRBTEST.COM for oryx/127.6.234.254@KRBTEST.COM, Server not found in Kerberos database
I20260620 11:43:01.868594 16206 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 11:43:01.863200 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.234.254:38223 (local address 127.6.234.195:47147)
0620 11:43:01.863355 (+ 155us) negotiation.cc:107] Waiting for socket to connect
0620 11:43:01.863367 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0620 11:43:01.863422 (+ 55us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 11:43:01.863579 (+ 157us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 11:43:01.863580 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 11:43:01.863732 (+ 152us) client_negotiation.cc:189] Negotiated authn=SASL
0620 11:43:01.863771 (+ 39us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 11:43:01.863772 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 11:43:01.864754 (+ 982us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 11:43:01.864755 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 11:43:01.865296 (+ 541us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 11:43:01.865298 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 11:43:01.865724 (+ 426us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 11:43:01.865725 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 11:43:01.865793 (+ 68us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 11:43:01.866477 (+ 684us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 11:43:01.866482 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0620 11:43:01.868487 (+ 2005us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.6.234.254:38223: Server oryx/127.6.234.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":132,"thread_start_us":94,"threads_started":1}
W20260620 11:43:01.868659 16207 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 11:43:01.863236 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.6.234.195:47147 (local address 127.6.234.254:38223)
0620 11:43:01.863358 (+ 122us) server_negotiation.cc:206] Beginning negotiation
0620 11:43:01.863360 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 11:43:01.863427 (+ 67us) server_negotiation.cc:407] Connection header received
0620 11:43:01.863482 (+ 55us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 11:43:01.863483 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 11:43:01.863494 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 11:43:01.863574 (+ 80us) server_negotiation.cc:226] Negotiated authn=SASL
0620 11:43:01.863866 (+ 292us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 11:43:01.864643 (+ 777us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 11:43:01.865382 (+ 739us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 11:43:01.865607 (+ 225us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 11:43:01.865679 (+ 72us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 11:43:01.868593 (+ 2914us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.6.234.195:47147: BlockingRecv error: recv got EOF from 127.6.234.195:47147 (error 108)
Metrics: {"server-negotiator.queue_time_us":108,"thread_start_us":68,"threads_started":1}
I20260620 11:43:01.870596 15940 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 11:43:01.863054 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40140 (local address 127.6.234.254:38223)
0620 11:43:01.863090 (+ 36us) server_negotiation.cc:206] Beginning negotiation
0620 11:43:01.863092 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 11:43:01.863169 (+ 77us) server_negotiation.cc:407] Connection header received
0620 11:43:01.863294 (+ 125us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 11:43:01.863296 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 11:43:01.863308 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 11:43:01.863367 (+ 59us) server_negotiation.cc:226] Negotiated authn=SASL
0620 11:43:01.863967 (+ 600us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 11:43:01.864633 (+ 666us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 11:43:01.865427 (+ 794us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 11:43:01.865621 (+ 194us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 11:43:01.865697 (+ 76us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 11:43:01.867138 (+ 1441us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 11:43:01.867143 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 11:43:01.867144 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 11:43:01.867156 (+ 12us) server_negotiation.cc:922] calling sasl_server_start()
0620 11:43:01.869212 (+ 2056us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 11:43:01.869570 (+ 358us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 11:43:01.869574 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 11:43:01.869574 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 11:43:01.869688 (+ 114us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 11:43:01.869901 (+ 213us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 11:43:01.869902 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 11:43:01.869903 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 11:43:01.870184 (+ 281us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 11:43:01.870318 (+ 134us) server_negotiation.cc:1035] Waiting for connection context
0620 11:43:01.870447 (+ 129us) server_negotiation.cc:299] Negotiation successful
0620 11:43:01.870526 (+ 79us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":18}
Jun 20 11:43:01 dist-test-slave-cjc6 krb5kdc[15206](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.6.234.195@KRBTEST.COM for oryx/127.6.234.254@KRBTEST.COM, Server not found in Kerberos database
Jun 20 11:43:01 dist-test-slave-cjc6 krb5kdc[15206](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.6.234.195@KRBTEST.COM for oryx/127.6.234.254@KRBTEST.COM, Server not found in Kerberos database
I20260620 11:43:01.873435 16206 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 11:43:01.868876 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.234.254:38223 (local address 127.6.234.195:51207)
0620 11:43:01.868917 (+ 41us) negotiation.cc:107] Waiting for socket to connect
0620 11:43:01.868926 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0620 11:43:01.868956 (+ 30us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 11:43:01.869100 (+ 144us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 11:43:01.869101 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 11:43:01.869267 (+ 166us) client_negotiation.cc:189] Negotiated authn=SASL
0620 11:43:01.869301 (+ 34us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 11:43:01.869303 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 11:43:01.870209 (+ 906us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 11:43:01.870210 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 11:43:01.871000 (+ 790us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 11:43:01.871002 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 11:43:01.871396 (+ 394us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 11:43:01.871396 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 11:43:01.871455 (+ 59us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 11:43:01.871855 (+ 400us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 11:43:01.871859 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0620 11:43:01.873357 (+ 1498us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.6.234.254:38223: Server oryx/127.6.234.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":30}
W20260620 11:43:01.873490 16207 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 11:43:01.868941 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.6.234.195:51207 (local address 127.6.234.254:38223)
0620 11:43:01.868985 (+ 44us) server_negotiation.cc:206] Beginning negotiation
0620 11:43:01.868987 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 11:43:01.868995 (+ 8us) server_negotiation.cc:407] Connection header received
0620 11:43:01.869015 (+ 20us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 11:43:01.869016 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 11:43:01.869024 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 11:43:01.869097 (+ 73us) server_negotiation.cc:226] Negotiated authn=SASL
0620 11:43:01.869388 (+ 291us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 11:43:01.870133 (+ 745us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 11:43:01.871083 (+ 950us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 11:43:01.871309 (+ 226us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 11:43:01.871395 (+ 86us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 11:43:01.873428 (+ 2033us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.6.234.195:51207: BlockingRecv error: recv got EOF from 127.6.234.195:51207 (error 108)
Metrics: {"server-negotiator.queue_time_us":22}
W20260620 11:43:01.873772 16202 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
Jun 20 11:43:02 dist-test-slave-cjc6 krb5kdc[15206](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.6.234.193@KRBTEST.COM for oryx/127.6.234.254@KRBTEST.COM, Server not found in Kerberos database
Jun 20 11:43:02 dist-test-slave-cjc6 krb5kdc[15206](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.6.234.193@KRBTEST.COM for oryx/127.6.234.254@KRBTEST.COM, Server not found in Kerberos database
I20260620 11:43:02.630031 16208 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 11:43:02.625154 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.234.254:38223 (local address 127.6.234.193:45279)
0620 11:43:02.625284 (+ 130us) negotiation.cc:107] Waiting for socket to connect
0620 11:43:02.625296 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0620 11:43:02.625356 (+ 60us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 11:43:02.625540 (+ 184us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 11:43:02.625540 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 11:43:02.625754 (+ 214us) client_negotiation.cc:189] Negotiated authn=SASL
0620 11:43:02.625808 (+ 54us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 11:43:02.625810 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 11:43:02.626560 (+ 750us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 11:43:02.626561 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 11:43:02.627143 (+ 582us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 11:43:02.627145 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 11:43:02.627560 (+ 415us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 11:43:02.627561 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 11:43:02.627625 (+ 64us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 11:43:02.628071 (+ 446us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 11:43:02.628076 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0620 11:43:02.629963 (+ 1887us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.6.234.254:38223: Server oryx/127.6.234.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":112,"thread_start_us":83,"threads_started":1}
W20260620 11:43:02.630169 16209 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 11:43:02.625286 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.6.234.193:45279 (local address 127.6.234.254:38223)
0620 11:43:02.625412 (+ 126us) server_negotiation.cc:206] Beginning negotiation
0620 11:43:02.625414 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 11:43:02.625427 (+ 13us) server_negotiation.cc:407] Connection header received
0620 11:43:02.625451 (+ 24us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 11:43:02.625451 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 11:43:02.625461 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 11:43:02.625528 (+ 67us) server_negotiation.cc:226] Negotiated authn=SASL
0620 11:43:02.625902 (+ 374us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 11:43:02.626467 (+ 565us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 11:43:02.627242 (+ 775us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 11:43:02.627470 (+ 228us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 11:43:02.627530 (+ 60us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 11:43:02.630067 (+ 2537us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.6.234.193:45279: BlockingRecv error: recv got EOF from 127.6.234.193:45279 (error 108)
Metrics: {"server-negotiator.queue_time_us":106,"thread_start_us":79,"threads_started":1}
Jun 20 11:43:02 dist-test-slave-cjc6 krb5kdc[15206](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.6.234.194@KRBTEST.COM for oryx/127.6.234.254@KRBTEST.COM, Server not found in Kerberos database
Jun 20 11:43:02 dist-test-slave-cjc6 krb5kdc[15206](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.6.234.194@KRBTEST.COM for oryx/127.6.234.254@KRBTEST.COM, Server not found in Kerberos database
I20260620 11:43:02.757411 16210 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 11:43:02.752532 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.234.254:38223 (local address 127.6.234.194:52823)
0620 11:43:02.752711 (+ 179us) negotiation.cc:107] Waiting for socket to connect
0620 11:43:02.752723 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0620 11:43:02.752817 (+ 94us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 11:43:02.752972 (+ 155us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 11:43:02.752973 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 11:43:02.753164 (+ 191us) client_negotiation.cc:189] Negotiated authn=SASL
0620 11:43:02.753216 (+ 52us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 11:43:02.753218 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 11:43:02.753952 (+ 734us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 11:43:02.753954 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 11:43:02.754511 (+ 557us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 11:43:02.754512 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 11:43:02.754900 (+ 388us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 11:43:02.754901 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 11:43:02.754959 (+ 58us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 11:43:02.755491 (+ 532us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 11:43:02.755496 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0620 11:43:02.757302 (+ 1806us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.6.234.254:38223: Server oryx/127.6.234.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":154,"thread_start_us":67,"threads_started":1}
W20260620 11:43:02.757462 16209 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 11:43:02.752590 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.6.234.194:52823 (local address 127.6.234.254:38223)
0620 11:43:02.752668 (+ 78us) server_negotiation.cc:206] Beginning negotiation
0620 11:43:02.752670 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 11:43:02.752795 (+ 125us) server_negotiation.cc:407] Connection header received
0620 11:43:02.752874 (+ 79us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 11:43:02.752875 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 11:43:02.752890 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 11:43:02.752965 (+ 75us) server_negotiation.cc:226] Negotiated authn=SASL
0620 11:43:02.753301 (+ 336us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 11:43:02.753869 (+ 568us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 11:43:02.754603 (+ 734us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 11:43:02.754819 (+ 216us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 11:43:02.754901 (+ 82us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 11:43:02.757393 (+ 2492us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.6.234.194:52823: BlockingRecv error: recv got EOF from 127.6.234.194:52823 (error 108)
Metrics: {"server-negotiator.queue_time_us":51}
Jun 20 11:43:02 dist-test-slave-cjc6 krb5kdc[15206](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.6.234.195@KRBTEST.COM for oryx/127.6.234.254@KRBTEST.COM, Server not found in Kerberos database
Jun 20 11:43:02 dist-test-slave-cjc6 krb5kdc[15206](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.6.234.195@KRBTEST.COM for oryx/127.6.234.254@KRBTEST.COM, Server not found in Kerberos database
I20260620 11:43:02.879179 16211 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 11:43:02.874120 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.234.254:38223 (local address 127.6.234.195:33993)
0620 11:43:02.874311 (+ 191us) negotiation.cc:107] Waiting for socket to connect
0620 11:43:02.874324 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0620 11:43:02.874416 (+ 92us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 11:43:02.874596 (+ 180us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 11:43:02.874596 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 11:43:02.874798 (+ 202us) client_negotiation.cc:189] Negotiated authn=SASL
0620 11:43:02.874849 (+ 51us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 11:43:02.874852 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 11:43:02.875596 (+ 744us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 11:43:02.875597 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 11:43:02.876181 (+ 584us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 11:43:02.876183 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 11:43:02.876618 (+ 435us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 11:43:02.876620 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 11:43:02.876686 (+ 66us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 11:43:02.877220 (+ 534us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 11:43:02.877226 (+ 6us) client_negotiation.cc:656] calling sasl_client_start()
0620 11:43:02.879063 (+ 1837us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.6.234.254:38223: Server oryx/127.6.234.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":171,"thread_start_us":95,"threads_started":1}
W20260620 11:43:02.879225 16209 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 11:43:02.874213 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.6.234.195:33993 (local address 127.6.234.254:38223)
0620 11:43:02.874293 (+ 80us) server_negotiation.cc:206] Beginning negotiation
0620 11:43:02.874295 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 11:43:02.874396 (+ 101us) server_negotiation.cc:407] Connection header received
0620 11:43:02.874491 (+ 95us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 11:43:02.874492 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 11:43:02.874506 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 11:43:02.874589 (+ 83us) server_negotiation.cc:226] Negotiated authn=SASL
0620 11:43:02.874951 (+ 362us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 11:43:02.875498 (+ 547us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 11:43:02.876298 (+ 800us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 11:43:02.876522 (+ 224us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 11:43:02.876616 (+ 94us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 11:43:02.879151 (+ 2535us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.6.234.195:33993: BlockingRecv error: recv got EOF from 127.6.234.195:33993 (error 108)
Metrics: {"server-negotiator.queue_time_us":49}
Jun 20 11:43:03 dist-test-slave-cjc6 krb5kdc[15206](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.6.234.193@KRBTEST.COM for oryx/127.6.234.254@KRBTEST.COM, Server not found in Kerberos database
Jun 20 11:43:03 dist-test-slave-cjc6 krb5kdc[15206](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.6.234.193@KRBTEST.COM for oryx/127.6.234.254@KRBTEST.COM, Server not found in Kerberos database
I20260620 11:43:03.635388 16212 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 11:43:03.630574 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.234.254:38223 (local address 127.6.234.193:49335)
0620 11:43:03.630757 (+ 183us) negotiation.cc:107] Waiting for socket to connect
0620 11:43:03.630768 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0620 11:43:03.630884 (+ 116us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 11:43:03.631050 (+ 166us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 11:43:03.631051 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 11:43:03.631212 (+ 161us) client_negotiation.cc:189] Negotiated authn=SASL
0620 11:43:03.631263 (+ 51us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 11:43:03.631265 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 11:43:03.631923 (+ 658us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 11:43:03.631925 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 11:43:03.632478 (+ 553us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 11:43:03.632480 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 11:43:03.632902 (+ 422us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 11:43:03.632903 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 11:43:03.632959 (+ 56us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 11:43:03.633436 (+ 477us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 11:43:03.633440 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0620 11:43:03.635328 (+ 1888us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.6.234.254:38223: Server oryx/127.6.234.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":162,"thread_start_us":107,"threads_started":1}
W20260620 11:43:03.635576 16213 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 11:43:03.630705 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.6.234.193:49335 (local address 127.6.234.254:38223)
0620 11:43:03.630838 (+ 133us) server_negotiation.cc:206] Beginning negotiation
0620 11:43:03.630840 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 11:43:03.630867 (+ 27us) server_negotiation.cc:407] Connection header received
0620 11:43:03.630958 (+ 91us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 11:43:03.630959 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 11:43:03.630970 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 11:43:03.631030 (+ 60us) server_negotiation.cc:226] Negotiated authn=SASL
0620 11:43:03.631336 (+ 306us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 11:43:03.631844 (+ 508us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 11:43:03.632569 (+ 725us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 11:43:03.632776 (+ 207us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 11:43:03.632842 (+ 66us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 11:43:03.635450 (+ 2608us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.6.234.193:49335: BlockingRecv error: recv got EOF from 127.6.234.193:49335 (error 108)
Metrics: {"server-negotiator.queue_time_us":108,"thread_start_us":81,"threads_started":1}
Jun 20 11:43:03 dist-test-slave-cjc6 krb5kdc[15206](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.6.234.194@KRBTEST.COM for oryx/127.6.234.254@KRBTEST.COM, Server not found in Kerberos database
Jun 20 11:43:03 dist-test-slave-cjc6 krb5kdc[15206](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.6.234.194@KRBTEST.COM for oryx/127.6.234.254@KRBTEST.COM, Server not found in Kerberos database
I20260620 11:43:03.762805 16214 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 11:43:03.757900 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.234.254:38223 (local address 127.6.234.194:56465)
0620 11:43:03.758085 (+ 185us) negotiation.cc:107] Waiting for socket to connect
0620 11:43:03.758097 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0620 11:43:03.758184 (+ 87us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 11:43:03.758342 (+ 158us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 11:43:03.758343 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 11:43:03.758549 (+ 206us) client_negotiation.cc:189] Negotiated authn=SASL
0620 11:43:03.758621 (+ 72us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 11:43:03.758624 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 11:43:03.759301 (+ 677us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 11:43:03.759302 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 11:43:03.759843 (+ 541us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 11:43:03.759845 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 11:43:03.760233 (+ 388us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 11:43:03.760234 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 11:43:03.760292 (+ 58us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 11:43:03.760800 (+ 508us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 11:43:03.760805 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0620 11:43:03.762709 (+ 1904us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.6.234.254:38223: Server oryx/127.6.234.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":167,"thread_start_us":100,"threads_started":1}
W20260620 11:43:03.762863 16213 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 11:43:03.757997 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.6.234.194:56465 (local address 127.6.234.254:38223)
0620 11:43:03.758080 (+ 83us) server_negotiation.cc:206] Beginning negotiation
0620 11:43:03.758082 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 11:43:03.758164 (+ 82us) server_negotiation.cc:407] Connection header received
0620 11:43:03.758248 (+ 84us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 11:43:03.758249 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 11:43:03.758265 (+ 16us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 11:43:03.758319 (+ 54us) server_negotiation.cc:226] Negotiated authn=SASL
0620 11:43:03.758706 (+ 387us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 11:43:03.759206 (+ 500us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 11:43:03.759943 (+ 737us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 11:43:03.760144 (+ 201us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 11:43:03.760234 (+ 90us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 11:43:03.762791 (+ 2557us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.6.234.194:56465: BlockingRecv error: recv got EOF from 127.6.234.194:56465 (error 108)
Metrics: {"server-negotiator.queue_time_us":44}
Jun 20 11:43:03 dist-test-slave-cjc6 krb5kdc[15206](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.6.234.195@KRBTEST.COM for oryx/127.6.234.254@KRBTEST.COM, Server not found in Kerberos database
Jun 20 11:43:03 dist-test-slave-cjc6 krb5kdc[15206](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.6.234.195@KRBTEST.COM for oryx/127.6.234.254@KRBTEST.COM, Server not found in Kerberos database
I20260620 11:43:03.884332 16215 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 11:43:03.879678 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.234.254:38223 (local address 127.6.234.195:53449)
0620 11:43:03.879861 (+ 183us) negotiation.cc:107] Waiting for socket to connect
0620 11:43:03.879872 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0620 11:43:03.879922 (+ 50us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 11:43:03.880111 (+ 189us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 11:43:03.880112 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 11:43:03.880281 (+ 169us) client_negotiation.cc:189] Negotiated authn=SASL
0620 11:43:03.880330 (+ 49us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 11:43:03.880332 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 11:43:03.881035 (+ 703us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 11:43:03.881037 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 11:43:03.881579 (+ 542us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 11:43:03.881581 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 11:43:03.882019 (+ 438us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 11:43:03.882020 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 11:43:03.882074 (+ 54us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 11:43:03.882547 (+ 473us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 11:43:03.882551 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0620 11:43:03.884272 (+ 1721us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.6.234.254:38223: Server oryx/127.6.234.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":159,"thread_start_us":121,"threads_started":1}
W20260620 11:43:03.884475 16213 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 11:43:03.879825 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.6.234.195:53449 (local address 127.6.234.254:38223)
0620 11:43:03.879892 (+ 67us) server_negotiation.cc:206] Beginning negotiation
0620 11:43:03.879894 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 11:43:03.879907 (+ 13us) server_negotiation.cc:407] Connection header received
0620 11:43:03.879982 (+ 75us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 11:43:03.879990 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 11:43:03.880004 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 11:43:03.880077 (+ 73us) server_negotiation.cc:226] Negotiated authn=SASL
0620 11:43:03.880425 (+ 348us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 11:43:03.880944 (+ 519us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 11:43:03.881739 (+ 795us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 11:43:03.881935 (+ 196us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 11:43:03.882012 (+ 77us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 11:43:03.884391 (+ 2379us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.6.234.195:53449: BlockingRecv error: recv got EOF from 127.6.234.195:53449 (error 108)
Metrics: {"server-negotiator.queue_time_us":43}
Jun 20 11:43:04 dist-test-slave-cjc6 krb5kdc[15206](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.6.234.193@KRBTEST.COM for oryx/127.6.234.254@KRBTEST.COM, Server not found in Kerberos database
Jun 20 11:43:04 dist-test-slave-cjc6 krb5kdc[15206](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.6.234.193@KRBTEST.COM for oryx/127.6.234.254@KRBTEST.COM, Server not found in Kerberos database
I20260620 11:43:04.644189 16216 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 11:43:04.635927 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.234.254:38223 (local address 127.6.234.193:54823)
0620 11:43:04.636063 (+ 136us) negotiation.cc:107] Waiting for socket to connect
0620 11:43:04.636074 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0620 11:43:04.636130 (+ 56us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 11:43:04.636309 (+ 179us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 11:43:04.636310 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 11:43:04.636464 (+ 154us) client_negotiation.cc:189] Negotiated authn=SASL
0620 11:43:04.636512 (+ 48us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 11:43:04.636513 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 11:43:04.637144 (+ 631us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 11:43:04.637145 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 11:43:04.637662 (+ 517us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 11:43:04.637664 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 11:43:04.638081 (+ 417us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 11:43:04.638082 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 11:43:04.638125 (+ 43us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 11:43:04.642352 (+ 4227us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 11:43:04.642358 (+ 6us) client_negotiation.cc:656] calling sasl_client_start()
0620 11:43:04.644130 (+ 1772us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.6.234.254:38223: Server oryx/127.6.234.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":114,"thread_start_us":85,"threads_started":1}
W20260620 11:43:04.644328 16217 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 11:43:04.636005 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.6.234.193:54823 (local address 127.6.234.254:38223)
0620 11:43:04.636171 (+ 166us) server_negotiation.cc:206] Beginning negotiation
0620 11:43:04.636173 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 11:43:04.636182 (+ 9us) server_negotiation.cc:407] Connection header received
0620 11:43:04.636218 (+ 36us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 11:43:04.636219 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 11:43:04.636230 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 11:43:04.636293 (+ 63us) server_negotiation.cc:226] Negotiated authn=SASL
0620 11:43:04.636576 (+ 283us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 11:43:04.637066 (+ 490us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 11:43:04.637812 (+ 746us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 11:43:04.638012 (+ 200us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 11:43:04.638078 (+ 66us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 11:43:04.644237 (+ 6159us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.6.234.193:54823: BlockingRecv error: recv got EOF from 127.6.234.193:54823 (error 108)
Metrics: {"server-negotiator.queue_time_us":141,"thread_start_us":106,"threads_started":1}
Jun 20 11:43:04 dist-test-slave-cjc6 krb5kdc[15206](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.6.234.194@KRBTEST.COM for oryx/127.6.234.254@KRBTEST.COM, Server not found in Kerberos database
Jun 20 11:43:04 dist-test-slave-cjc6 krb5kdc[15206](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.6.234.194@KRBTEST.COM for oryx/127.6.234.254@KRBTEST.COM, Server not found in Kerberos database
I20260620 11:43:04.768256 16218 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 11:43:04.763299 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.234.254:38223 (local address 127.6.234.194:37181)
0620 11:43:04.763437 (+ 138us) negotiation.cc:107] Waiting for socket to connect
0620 11:43:04.763447 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0620 11:43:04.763504 (+ 57us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 11:43:04.763619 (+ 115us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 11:43:04.763620 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 11:43:04.763777 (+ 157us) client_negotiation.cc:189] Negotiated authn=SASL
0620 11:43:04.763823 (+ 46us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 11:43:04.763825 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 11:43:04.764540 (+ 715us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 11:43:04.764541 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 11:43:04.765064 (+ 523us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 11:43:04.765066 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 11:43:04.765472 (+ 406us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 11:43:04.765473 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 11:43:04.765524 (+ 51us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 11:43:04.765987 (+ 463us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 11:43:04.765992 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0620 11:43:04.768152 (+ 2160us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.6.234.254:38223: Server oryx/127.6.234.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":93,"thread_start_us":66,"threads_started":1}
W20260620 11:43:04.768362 16217 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 11:43:04.763344 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.6.234.194:37181 (local address 127.6.234.254:38223)
0620 11:43:04.763390 (+ 46us) server_negotiation.cc:206] Beginning negotiation
0620 11:43:04.763393 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0620 11:43:04.763512 (+ 119us) server_negotiation.cc:407] Connection header received
0620 11:43:04.763554 (+ 42us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 11:43:04.763555 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 11:43:04.763566 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 11:43:04.763608 (+ 42us) server_negotiation.cc:226] Negotiated authn=SASL
0620 11:43:04.763914 (+ 306us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 11:43:04.764426 (+ 512us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 11:43:04.765164 (+ 738us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 11:43:04.765376 (+ 212us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 11:43:04.765444 (+ 68us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 11:43:04.768315 (+ 2871us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.6.234.194:37181: BlockingRecv error: recv got EOF from 127.6.234.194:37181 (error 108)
Metrics: {"server-negotiator.queue_time_us":22}
Jun 20 11:43:04 dist-test-slave-cjc6 krb5kdc[15206](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.6.234.195@KRBTEST.COM for oryx/127.6.234.254@KRBTEST.COM, Server not found in Kerberos database
Jun 20 11:43:04 dist-test-slave-cjc6 krb5kdc[15206](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.6.234.195@KRBTEST.COM for oryx/127.6.234.254@KRBTEST.COM, Server not found in Kerberos database
I20260620 11:43:04.889683 16219 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 11:43:04.884955 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.234.254:38223 (local address 127.6.234.195:39759)
0620 11:43:04.885119 (+ 164us) negotiation.cc:107] Waiting for socket to connect
0620 11:43:04.885132 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0620 11:43:04.885208 (+ 76us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 11:43:04.885366 (+ 158us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 11:43:04.885366 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 11:43:04.885525 (+ 159us) client_negotiation.cc:189] Negotiated authn=SASL
0620 11:43:04.885574 (+ 49us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 11:43:04.885576 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 11:43:04.886366 (+ 790us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 11:43:04.886367 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 11:43:04.886908 (+ 541us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 11:43:04.886910 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 11:43:04.887302 (+ 392us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 11:43:04.887303 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 11:43:04.887421 (+ 118us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 11:43:04.887876 (+ 455us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 11:43:04.887881 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0620 11:43:04.889621 (+ 1740us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.6.234.254:38223: Server oryx/127.6.234.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":143,"thread_start_us":92,"threads_started":1}
W20260620 11:43:04.889890 16217 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 11:43:04.885036 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.6.234.195:39759 (local address 127.6.234.254:38223)
0620 11:43:04.885112 (+ 76us) server_negotiation.cc:206] Beginning negotiation
0620 11:43:04.885114 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 11:43:04.885215 (+ 101us) server_negotiation.cc:407] Connection header received
0620 11:43:04.885266 (+ 51us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 11:43:04.885267 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 11:43:04.885283 (+ 16us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 11:43:04.885355 (+ 72us) server_negotiation.cc:226] Negotiated authn=SASL
0620 11:43:04.885676 (+ 321us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 11:43:04.886286 (+ 610us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 11:43:04.887006 (+ 720us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 11:43:04.887204 (+ 198us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 11:43:04.887273 (+ 69us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 11:43:04.889804 (+ 2531us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.6.234.195:39759: BlockingRecv error: recv got EOF from 127.6.234.195:39759 (error 108)
Metrics: {"server-negotiator.queue_time_us":44}
Jun 20 11:43:05 dist-test-slave-cjc6 krb5kdc[15206](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.6.234.193@KRBTEST.COM for oryx/127.6.234.254@KRBTEST.COM, Server not found in Kerberos database
Jun 20 11:43:05 dist-test-slave-cjc6 krb5kdc[15206](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.6.234.193@KRBTEST.COM for oryx/127.6.234.254@KRBTEST.COM, Server not found in Kerberos database
I20260620 11:43:05.649303 16220 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 11:43:05.644729 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.234.254:38223 (local address 127.6.234.193:54101)
0620 11:43:05.644873 (+ 144us) negotiation.cc:107] Waiting for socket to connect
0620 11:43:05.644885 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0620 11:43:05.644941 (+ 56us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 11:43:05.645093 (+ 152us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 11:43:05.645093 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 11:43:05.645254 (+ 161us) client_negotiation.cc:189] Negotiated authn=SASL
0620 11:43:05.645303 (+ 49us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 11:43:05.645305 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 11:43:05.646036 (+ 731us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 11:43:05.646037 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 11:43:05.646585 (+ 548us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 11:43:05.646586 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 11:43:05.646913 (+ 327us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 11:43:05.646914 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 11:43:05.646955 (+ 41us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 11:43:05.647414 (+ 459us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 11:43:05.647419 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0620 11:43:05.649242 (+ 1823us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.6.234.254:38223: Server oryx/127.6.234.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":125,"thread_start_us":90,"threads_started":1}
W20260620 11:43:05.649433 16221 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 11:43:05.644808 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.6.234.193:54101 (local address 127.6.234.254:38223)
0620 11:43:05.644967 (+ 159us) server_negotiation.cc:206] Beginning negotiation
0620 11:43:05.644969 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 11:43:05.644978 (+ 9us) server_negotiation.cc:407] Connection header received
0620 11:43:05.644999 (+ 21us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 11:43:05.645000 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 11:43:05.645010 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 11:43:05.645075 (+ 65us) server_negotiation.cc:226] Negotiated authn=SASL
0620 11:43:05.645374 (+ 299us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 11:43:05.645942 (+ 568us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 11:43:05.646663 (+ 721us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 11:43:05.646850 (+ 187us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 11:43:05.646901 (+ 51us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 11:43:05.649348 (+ 2447us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.6.234.193:54101: BlockingRecv error: recv got EOF from 127.6.234.193:54101 (error 108)
Metrics: {"server-negotiator.queue_time_us":133,"thread_start_us":94,"threads_started":1}
Jun 20 11:43:05 dist-test-slave-cjc6 krb5kdc[15206](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.6.234.194@KRBTEST.COM for oryx/127.6.234.254@KRBTEST.COM, Server not found in Kerberos database
Jun 20 11:43:05 dist-test-slave-cjc6 krb5kdc[15206](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.6.234.194@KRBTEST.COM for oryx/127.6.234.254@KRBTEST.COM, Server not found in Kerberos database
I20260620 11:43:05.774003 16222 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 11:43:05.768948 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.234.254:38223 (local address 127.6.234.194:52117)
0620 11:43:05.769174 (+ 226us) negotiation.cc:107] Waiting for socket to connect
0620 11:43:05.769187 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0620 11:43:05.769281 (+ 94us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 11:43:05.769458 (+ 177us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 11:43:05.769459 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 11:43:05.769640 (+ 181us) client_negotiation.cc:189] Negotiated authn=SASL
0620 11:43:05.769686 (+ 46us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 11:43:05.769688 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 11:43:05.770393 (+ 705us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 11:43:05.770394 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 11:43:05.770938 (+ 544us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 11:43:05.770940 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 11:43:05.771357 (+ 417us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 11:43:05.771358 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 11:43:05.771423 (+ 65us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 11:43:05.771876 (+ 453us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 11:43:05.771880 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0620 11:43:05.773943 (+ 2063us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.6.234.254:38223: Server oryx/127.6.234.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":201,"thread_start_us":112,"threads_started":1}
W20260620 11:43:05.774137 16221 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 11:43:05.769111 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.6.234.194:52117 (local address 127.6.234.254:38223)
0620 11:43:05.769179 (+ 68us) server_negotiation.cc:206] Beginning negotiation
0620 11:43:05.769182 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0620 11:43:05.769259 (+ 77us) server_negotiation.cc:407] Connection header received
0620 11:43:05.769351 (+ 92us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 11:43:05.769353 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 11:43:05.769365 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 11:43:05.769438 (+ 73us) server_negotiation.cc:226] Negotiated authn=SASL
0620 11:43:05.769792 (+ 354us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 11:43:05.770292 (+ 500us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 11:43:05.771045 (+ 753us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 11:43:05.771245 (+ 200us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 11:43:05.771340 (+ 95us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 11:43:05.774055 (+ 2715us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.6.234.194:52117: BlockingRecv error: recv got EOF from 127.6.234.194:52117 (error 108)
Metrics: {"server-negotiator.queue_time_us":44}
Jun 20 11:43:05 dist-test-slave-cjc6 krb5kdc[15206](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.6.234.195@KRBTEST.COM for oryx/127.6.234.254@KRBTEST.COM, Server not found in Kerberos database
Jun 20 11:43:05 dist-test-slave-cjc6 krb5kdc[15206](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.6.234.195@KRBTEST.COM for oryx/127.6.234.254@KRBTEST.COM, Server not found in Kerberos database
I20260620 11:43:05.894917 16223 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 11:43:05.890194 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.234.254:38223 (local address 127.6.234.195:46189)
0620 11:43:05.890393 (+ 199us) negotiation.cc:107] Waiting for socket to connect
0620 11:43:05.890406 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0620 11:43:05.890493 (+ 87us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 11:43:05.890656 (+ 163us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 11:43:05.890657 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 11:43:05.890852 (+ 195us) client_negotiation.cc:189] Negotiated authn=SASL
0620 11:43:05.890900 (+ 48us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 11:43:05.890901 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 11:43:05.891586 (+ 685us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 11:43:05.891587 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 11:43:05.892192 (+ 605us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 11:43:05.892194 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 11:43:05.892540 (+ 346us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 11:43:05.892541 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 11:43:05.892596 (+ 55us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 11:43:05.893028 (+ 432us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 11:43:05.893033 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0620 11:43:05.894820 (+ 1787us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.6.234.254:38223: Server oryx/127.6.234.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":180,"thread_start_us":88,"threads_started":1}
W20260620 11:43:05.894963 16221 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 11:43:05.890305 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.6.234.195:46189 (local address 127.6.234.254:38223)
0620 11:43:05.890377 (+ 72us) server_negotiation.cc:206] Beginning negotiation
0620 11:43:05.890379 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 11:43:05.890472 (+ 93us) server_negotiation.cc:407] Connection header received
0620 11:43:05.890558 (+ 86us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 11:43:05.890559 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 11:43:05.890574 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 11:43:05.890647 (+ 73us) server_negotiation.cc:226] Negotiated authn=SASL
0620 11:43:05.890994 (+ 347us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 11:43:05.891490 (+ 496us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 11:43:05.892274 (+ 784us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 11:43:05.892464 (+ 190us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 11:43:05.892542 (+ 78us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 11:43:05.894897 (+ 2355us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.6.234.195:46189: BlockingRecv error: recv got EOF from 127.6.234.195:46189 (error 108)
Metrics: {"server-negotiator.queue_time_us":41}
Jun 20 11:43:06 dist-test-slave-cjc6 krb5kdc[15206](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.6.234.193@KRBTEST.COM for oryx/127.6.234.254@KRBTEST.COM, Server not found in Kerberos database
Jun 20 11:43:06 dist-test-slave-cjc6 krb5kdc[15206](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.6.234.193@KRBTEST.COM for oryx/127.6.234.254@KRBTEST.COM, Server not found in Kerberos database
I20260620 11:43:06.654398 16224 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 11:43:06.649885 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.234.254:38223 (local address 127.6.234.193:35533)
0620 11:43:06.650062 (+ 177us) negotiation.cc:107] Waiting for socket to connect
0620 11:43:06.650073 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0620 11:43:06.650128 (+ 55us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 11:43:06.650274 (+ 146us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 11:43:06.650274 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 11:43:06.650429 (+ 155us) client_negotiation.cc:189] Negotiated authn=SASL
0620 11:43:06.650476 (+ 47us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 11:43:06.650477 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 11:43:06.651119 (+ 642us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 11:43:06.651120 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 11:43:06.651627 (+ 507us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 11:43:06.651628 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 11:43:06.651963 (+ 335us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 11:43:06.651963 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 11:43:06.652000 (+ 37us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 11:43:06.652425 (+ 425us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 11:43:06.652429 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0620 11:43:06.654338 (+ 1909us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.6.234.254:38223: Server oryx/127.6.234.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":156,"thread_start_us":116,"threads_started":1}
W20260620 11:43:06.654546 16225 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 11:43:06.650005 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.6.234.193:35533 (local address 127.6.234.254:38223)
0620 11:43:06.650154 (+ 149us) server_negotiation.cc:206] Beginning negotiation
0620 11:43:06.650156 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 11:43:06.650165 (+ 9us) server_negotiation.cc:407] Connection header received
0620 11:43:06.650189 (+ 24us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 11:43:06.650189 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 11:43:06.650199 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 11:43:06.650261 (+ 62us) server_negotiation.cc:226] Negotiated authn=SASL
0620 11:43:06.650542 (+ 281us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 11:43:06.651055 (+ 513us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 11:43:06.651696 (+ 641us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 11:43:06.651891 (+ 195us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 11:43:06.651939 (+ 48us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 11:43:06.654455 (+ 2516us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.6.234.193:35533: BlockingRecv error: recv got EOF from 127.6.234.193:35533 (error 108)
Metrics: {"server-negotiator.queue_time_us":124,"thread_start_us":91,"threads_started":1}
Jun 20 11:43:06 dist-test-slave-cjc6 krb5kdc[15206](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.6.234.194@KRBTEST.COM for oryx/127.6.234.254@KRBTEST.COM, Server not found in Kerberos database
Jun 20 11:43:06 dist-test-slave-cjc6 krb5kdc[15206](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.6.234.194@KRBTEST.COM for oryx/127.6.234.254@KRBTEST.COM, Server not found in Kerberos database
I20260620 11:43:06.779060 16226 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 11:43:06.774443 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.234.254:38223 (local address 127.6.234.194:51059)
0620 11:43:06.774592 (+ 149us) negotiation.cc:107] Waiting for socket to connect
0620 11:43:06.774602 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0620 11:43:06.774649 (+ 47us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 11:43:06.774773 (+ 124us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 11:43:06.774773 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 11:43:06.774939 (+ 166us) client_negotiation.cc:189] Negotiated authn=SASL
0620 11:43:06.774987 (+ 48us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 11:43:06.774989 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 11:43:06.775643 (+ 654us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 11:43:06.775643 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 11:43:06.776166 (+ 523us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 11:43:06.776168 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 11:43:06.776492 (+ 324us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 11:43:06.776493 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 11:43:06.776532 (+ 39us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 11:43:06.777156 (+ 624us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 11:43:06.777161 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0620 11:43:06.779000 (+ 1839us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.6.234.254:38223: Server oryx/127.6.234.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":124,"thread_start_us":87,"threads_started":1}
W20260620 11:43:06.779172 16225 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 11:43:06.774582 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.6.234.194:51059 (local address 127.6.234.254:38223)
0620 11:43:06.774645 (+ 63us) server_negotiation.cc:206] Beginning negotiation
0620 11:43:06.774647 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 11:43:06.774656 (+ 9us) server_negotiation.cc:407] Connection header received
0620 11:43:06.774693 (+ 37us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 11:43:06.774694 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 11:43:06.774704 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 11:43:06.774757 (+ 53us) server_negotiation.cc:226] Negotiated authn=SASL
0620 11:43:06.775079 (+ 322us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 11:43:06.775571 (+ 492us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 11:43:06.776240 (+ 669us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 11:43:06.776431 (+ 191us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 11:43:06.776482 (+ 51us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 11:43:06.779099 (+ 2617us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.6.234.194:51059: BlockingRecv error: recv got EOF from 127.6.234.194:51059 (error 108)
Metrics: {"server-negotiator.queue_time_us":40}
Jun 20 11:43:06 dist-test-slave-cjc6 krb5kdc[15206](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.6.234.195@KRBTEST.COM for oryx/127.6.234.254@KRBTEST.COM, Server not found in Kerberos database
Jun 20 11:43:06 dist-test-slave-cjc6 krb5kdc[15206](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.6.234.195@KRBTEST.COM for oryx/127.6.234.254@KRBTEST.COM, Server not found in Kerberos database
I20260620 11:43:06.899853 16227 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 11:43:06.895407 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.234.254:38223 (local address 127.6.234.195:55811)
0620 11:43:06.895549 (+ 142us) negotiation.cc:107] Waiting for socket to connect
0620 11:43:06.895568 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0620 11:43:06.895623 (+ 55us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 11:43:06.895761 (+ 138us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 11:43:06.895762 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 11:43:06.895941 (+ 179us) client_negotiation.cc:189] Negotiated authn=SASL
0620 11:43:06.895987 (+ 46us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 11:43:06.895989 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 11:43:06.896665 (+ 676us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 11:43:06.896666 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 11:43:06.897199 (+ 533us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 11:43:06.897201 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 11:43:06.897554 (+ 353us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 11:43:06.897555 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 11:43:06.897595 (+ 40us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 11:43:06.898086 (+ 491us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 11:43:06.898091 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0620 11:43:06.899796 (+ 1705us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.6.234.254:38223: Server oryx/127.6.234.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":120,"thread_start_us":61,"threads_started":1}
W20260620 11:43:06.899987 16225 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 11:43:06.895498 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.6.234.195:55811 (local address 127.6.234.254:38223)
0620 11:43:06.895618 (+ 120us) server_negotiation.cc:206] Beginning negotiation
0620 11:43:06.895620 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 11:43:06.895630 (+ 10us) server_negotiation.cc:407] Connection header received
0620 11:43:06.895672 (+ 42us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 11:43:06.895672 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 11:43:06.895687 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 11:43:06.895739 (+ 52us) server_negotiation.cc:226] Negotiated authn=SASL
0620 11:43:06.896067 (+ 328us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 11:43:06.896589 (+ 522us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 11:43:06.897294 (+ 705us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 11:43:06.897481 (+ 187us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 11:43:06.897544 (+ 63us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 11:43:06.899899 (+ 2355us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.6.234.195:55811: BlockingRecv error: recv got EOF from 127.6.234.195:55811 (error 108)
Metrics: {"server-negotiator.queue_time_us":93}
Jun 20 11:43:07 dist-test-slave-cjc6 krb5kdc[15206](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.6.234.193@KRBTEST.COM for oryx/127.6.234.254@KRBTEST.COM, Server not found in Kerberos database
Jun 20 11:43:07 dist-test-slave-cjc6 krb5kdc[15206](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.6.234.193@KRBTEST.COM for oryx/127.6.234.254@KRBTEST.COM, Server not found in Kerberos database
I20260620 11:43:07.659529 16228 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 11:43:07.654890 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.234.254:38223 (local address 127.6.234.193:45875)
0620 11:43:07.655072 (+ 182us) negotiation.cc:107] Waiting for socket to connect
0620 11:43:07.655084 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0620 11:43:07.655140 (+ 56us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 11:43:07.655325 (+ 185us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 11:43:07.655326 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 11:43:07.655487 (+ 161us) client_negotiation.cc:189] Negotiated authn=SASL
0620 11:43:07.655536 (+ 49us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 11:43:07.655538 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 11:43:07.656180 (+ 642us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 11:43:07.656182 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 11:43:07.656719 (+ 537us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 11:43:07.656721 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 11:43:07.657094 (+ 373us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 11:43:07.657094 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 11:43:07.657132 (+ 38us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 11:43:07.657576 (+ 444us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 11:43:07.657580 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0620 11:43:07.659432 (+ 1852us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.6.234.254:38223: Server oryx/127.6.234.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":163,"thread_start_us":107,"threads_started":1}
W20260620 11:43:07.659595 16229 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 11:43:07.654986 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.6.234.193:45875 (local address 127.6.234.254:38223)
0620 11:43:07.655109 (+ 123us) server_negotiation.cc:206] Beginning negotiation
0620 11:43:07.655111 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 11:43:07.655122 (+ 11us) server_negotiation.cc:407] Connection header received
0620 11:43:07.655205 (+ 83us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 11:43:07.655206 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 11:43:07.655217 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 11:43:07.655276 (+ 59us) server_negotiation.cc:226] Negotiated authn=SASL
0620 11:43:07.655605 (+ 329us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 11:43:07.656105 (+ 500us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 11:43:07.656821 (+ 716us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 11:43:07.657022 (+ 201us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 11:43:07.657086 (+ 64us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 11:43:07.659518 (+ 2432us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.6.234.193:45875: BlockingRecv error: recv got EOF from 127.6.234.193:45875 (error 108)
Metrics: {"server-negotiator.queue_time_us":104,"thread_start_us":55,"threads_started":1}
Jun 20 11:43:07 dist-test-slave-cjc6 krb5kdc[15206](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.6.234.194@KRBTEST.COM for oryx/127.6.234.254@KRBTEST.COM, Server not found in Kerberos database
Jun 20 11:43:07 dist-test-slave-cjc6 krb5kdc[15206](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.6.234.194@KRBTEST.COM for oryx/127.6.234.254@KRBTEST.COM, Server not found in Kerberos database
I20260620 11:43:07.784472 16230 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 11:43:07.779635 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.234.254:38223 (local address 127.6.234.194:45357)
0620 11:43:07.779826 (+ 191us) negotiation.cc:107] Waiting for socket to connect
0620 11:43:07.779845 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0620 11:43:07.779918 (+ 73us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 11:43:07.780078 (+ 160us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 11:43:07.780079 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 11:43:07.780248 (+ 169us) client_negotiation.cc:189] Negotiated authn=SASL
0620 11:43:07.780294 (+ 46us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 11:43:07.780296 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 11:43:07.780969 (+ 673us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 11:43:07.780970 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 11:43:07.781524 (+ 554us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 11:43:07.781526 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 11:43:07.781913 (+ 387us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 11:43:07.781913 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 11:43:07.781959 (+ 46us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 11:43:07.782488 (+ 529us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 11:43:07.782493 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0620 11:43:07.784362 (+ 1869us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.6.234.254:38223: Server oryx/127.6.234.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":166,"thread_start_us":111,"threads_started":1}
W20260620 11:43:07.784551 16229 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 11:43:07.779687 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.6.234.194:45357 (local address 127.6.234.254:38223)
0620 11:43:07.779759 (+ 72us) server_negotiation.cc:206] Beginning negotiation
0620 11:43:07.779761 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 11:43:07.779963 (+ 202us) server_negotiation.cc:407] Connection header received
0620 11:43:07.779998 (+ 35us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 11:43:07.779999 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 11:43:07.780012 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 11:43:07.780056 (+ 44us) server_negotiation.cc:226] Negotiated authn=SASL
0620 11:43:07.780370 (+ 314us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 11:43:07.780887 (+ 517us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 11:43:07.781611 (+ 724us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 11:43:07.781843 (+ 232us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 11:43:07.781900 (+ 57us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 11:43:07.784459 (+ 2559us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.6.234.194:45357: BlockingRecv error: recv got EOF from 127.6.234.194:45357 (error 108)
Metrics: {"server-negotiator.queue_time_us":42}
Jun 20 11:43:07 dist-test-slave-cjc6 krb5kdc[15206](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.6.234.195@KRBTEST.COM for oryx/127.6.234.254@KRBTEST.COM, Server not found in Kerberos database
Jun 20 11:43:07 dist-test-slave-cjc6 krb5kdc[15206](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.6.234.195@KRBTEST.COM for oryx/127.6.234.254@KRBTEST.COM, Server not found in Kerberos database
I20260620 11:43:07.905402 16231 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 11:43:07.900459 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.234.254:38223 (local address 127.6.234.195:50453)
0620 11:43:07.900678 (+ 219us) negotiation.cc:107] Waiting for socket to connect
0620 11:43:07.900690 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0620 11:43:07.900783 (+ 93us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 11:43:07.900956 (+ 173us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 11:43:07.900957 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 11:43:07.901141 (+ 184us) client_negotiation.cc:189] Negotiated authn=SASL
0620 11:43:07.901205 (+ 64us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 11:43:07.901207 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 11:43:07.901950 (+ 743us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 11:43:07.901951 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 11:43:07.902484 (+ 533us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 11:43:07.902486 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 11:43:07.902884 (+ 398us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 11:43:07.902885 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 11:43:07.902950 (+ 65us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 11:43:07.903385 (+ 435us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 11:43:07.903390 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0620 11:43:07.905342 (+ 1952us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.6.234.254:38223: Server oryx/127.6.234.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":201,"thread_start_us":103,"threads_started":1}
W20260620 11:43:07.905480 16229 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 11:43:07.900586 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.6.234.195:50453 (local address 127.6.234.254:38223)
0620 11:43:07.900652 (+ 66us) server_negotiation.cc:206] Beginning negotiation
0620 11:43:07.900654 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 11:43:07.900768 (+ 114us) server_negotiation.cc:407] Connection header received
0620 11:43:07.900882 (+ 114us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 11:43:07.900883 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 11:43:07.900898 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 11:43:07.900950 (+ 52us) server_negotiation.cc:226] Negotiated authn=SASL
0620 11:43:07.901290 (+ 340us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 11:43:07.901867 (+ 577us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 11:43:07.902578 (+ 711us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 11:43:07.902793 (+ 215us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 11:43:07.902880 (+ 87us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 11:43:07.905433 (+ 2553us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.6.234.195:50453: BlockingRecv error: recv got EOF from 127.6.234.195:50453 (error 108)
Metrics: {"server-negotiator.queue_time_us":26}
Jun 20 11:43:08 dist-test-slave-cjc6 krb5kdc[15206](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.6.234.193@KRBTEST.COM for oryx/127.6.234.254@KRBTEST.COM, Server not found in Kerberos database
Jun 20 11:43:08 dist-test-slave-cjc6 krb5kdc[15206](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.6.234.193@KRBTEST.COM for oryx/127.6.234.254@KRBTEST.COM, Server not found in Kerberos database
I20260620 11:43:08.664736 16232 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 11:43:08.660125 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.234.254:38223 (local address 127.6.234.193:42817)
0620 11:43:08.660275 (+ 150us) negotiation.cc:107] Waiting for socket to connect
0620 11:43:08.660287 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0620 11:43:08.660335 (+ 48us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 11:43:08.660453 (+ 118us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 11:43:08.660453 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 11:43:08.660620 (+ 167us) client_negotiation.cc:189] Negotiated authn=SASL
0620 11:43:08.660667 (+ 47us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 11:43:08.660669 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 11:43:08.661326 (+ 657us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 11:43:08.661328 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 11:43:08.661968 (+ 640us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 11:43:08.661970 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 11:43:08.662358 (+ 388us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 11:43:08.662359 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 11:43:08.662403 (+ 44us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 11:43:08.662877 (+ 474us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 11:43:08.662882 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0620 11:43:08.664666 (+ 1784us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.6.234.254:38223: Server oryx/127.6.234.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":127,"thread_start_us":79,"threads_started":1}
W20260620 11:43:08.664896 16233 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 11:43:08.660176 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.6.234.193:42817 (local address 127.6.234.254:38223)
0620 11:43:08.660315 (+ 139us) server_negotiation.cc:206] Beginning negotiation
0620 11:43:08.660317 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 11:43:08.660326 (+ 9us) server_negotiation.cc:407] Connection header received
0620 11:43:08.660381 (+ 55us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 11:43:08.660382 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 11:43:08.660392 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 11:43:08.660448 (+ 56us) server_negotiation.cc:226] Negotiated authn=SASL
0620 11:43:08.660740 (+ 292us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 11:43:08.661241 (+ 501us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 11:43:08.662077 (+ 836us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 11:43:08.662282 (+ 205us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 11:43:08.662347 (+ 65us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 11:43:08.664784 (+ 2437us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.6.234.193:42817: BlockingRecv error: recv got EOF from 127.6.234.193:42817 (error 108)
Metrics: {"server-negotiator.queue_time_us":119,"thread_start_us":91,"threads_started":1}
Jun 20 11:43:08 dist-test-slave-cjc6 krb5kdc[15206](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.6.234.194@KRBTEST.COM for oryx/127.6.234.254@KRBTEST.COM, Server not found in Kerberos database
Jun 20 11:43:08 dist-test-slave-cjc6 krb5kdc[15206](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.6.234.194@KRBTEST.COM for oryx/127.6.234.254@KRBTEST.COM, Server not found in Kerberos database
I20260620 11:43:08.789912 16234 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 11:43:08.785040 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.234.254:38223 (local address 127.6.234.194:42327)
0620 11:43:08.785232 (+ 192us) negotiation.cc:107] Waiting for socket to connect
0620 11:43:08.785243 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0620 11:43:08.785324 (+ 81us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 11:43:08.785479 (+ 155us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 11:43:08.785480 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 11:43:08.785669 (+ 189us) client_negotiation.cc:189] Negotiated authn=SASL
0620 11:43:08.785778 (+ 109us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 11:43:08.785781 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 11:43:08.786460 (+ 679us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 11:43:08.786462 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 11:43:08.787007 (+ 545us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 11:43:08.787009 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 11:43:08.787387 (+ 378us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 11:43:08.787388 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 11:43:08.787447 (+ 59us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 11:43:08.787957 (+ 510us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 11:43:08.787962 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0620 11:43:08.789819 (+ 1857us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.6.234.254:38223: Server oryx/127.6.234.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":178,"thread_start_us":102,"threads_started":1}
W20260620 11:43:08.789976 16233 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 11:43:08.785108 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.6.234.194:42327 (local address 127.6.234.254:38223)
0620 11:43:08.785184 (+ 76us) server_negotiation.cc:206] Beginning negotiation
0620 11:43:08.785187 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0620 11:43:08.785308 (+ 121us) server_negotiation.cc:407] Connection header received
0620 11:43:08.785384 (+ 76us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 11:43:08.785385 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 11:43:08.785399 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 11:43:08.785471 (+ 72us) server_negotiation.cc:226] Negotiated authn=SASL
0620 11:43:08.785865 (+ 394us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 11:43:08.786369 (+ 504us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 11:43:08.787102 (+ 733us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 11:43:08.787306 (+ 204us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 11:43:08.787387 (+ 81us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 11:43:08.789913 (+ 2526us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.6.234.194:42327: BlockingRecv error: recv got EOF from 127.6.234.194:42327 (error 108)
Metrics: {"server-negotiator.queue_time_us":44}
Jun 20 11:43:08 dist-test-slave-cjc6 krb5kdc[15206](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.6.234.195@KRBTEST.COM for oryx/127.6.234.254@KRBTEST.COM, Server not found in Kerberos database
Jun 20 11:43:08 dist-test-slave-cjc6 krb5kdc[15206](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.6.234.195@KRBTEST.COM for oryx/127.6.234.254@KRBTEST.COM, Server not found in Kerberos database
I20260620 11:43:08.910789 16235 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 11:43:08.905907 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.234.254:38223 (local address 127.6.234.195:53565)
0620 11:43:08.906072 (+ 165us) negotiation.cc:107] Waiting for socket to connect
0620 11:43:08.906087 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0620 11:43:08.906160 (+ 73us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 11:43:08.906287 (+ 127us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 11:43:08.906288 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 11:43:08.906462 (+ 174us) client_negotiation.cc:189] Negotiated authn=SASL
0620 11:43:08.906510 (+ 48us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 11:43:08.906512 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 11:43:08.907192 (+ 680us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 11:43:08.907193 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 11:43:08.907729 (+ 536us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 11:43:08.907731 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 11:43:08.908135 (+ 404us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 11:43:08.908136 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 11:43:08.908200 (+ 64us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 11:43:08.908785 (+ 585us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 11:43:08.908791 (+ 6us) client_negotiation.cc:656] calling sasl_client_start()
0620 11:43:08.910686 (+ 1895us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.6.234.254:38223: Server oryx/127.6.234.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":122,"thread_start_us":95,"threads_started":1}
W20260620 11:43:08.910838 16233 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 11:43:08.906015 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.6.234.195:53565 (local address 127.6.234.254:38223)
0620 11:43:08.906088 (+ 73us) server_negotiation.cc:206] Beginning negotiation
0620 11:43:08.906090 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 11:43:08.906154 (+ 64us) server_negotiation.cc:407] Connection header received
0620 11:43:08.906217 (+ 63us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 11:43:08.906218 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 11:43:08.906233 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 11:43:08.906284 (+ 51us) server_negotiation.cc:226] Negotiated authn=SASL
0620 11:43:08.906591 (+ 307us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 11:43:08.907102 (+ 511us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 11:43:08.907816 (+ 714us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 11:43:08.908045 (+ 229us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 11:43:08.908131 (+ 86us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 11:43:08.910772 (+ 2641us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.6.234.195:53565: BlockingRecv error: recv got EOF from 127.6.234.195:53565 (error 108)
Metrics: {"server-negotiator.queue_time_us":49}
Jun 20 11:43:09 dist-test-slave-cjc6 krb5kdc[15206](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.6.234.193@KRBTEST.COM for oryx/127.6.234.254@KRBTEST.COM, Server not found in Kerberos database
Jun 20 11:43:09 dist-test-slave-cjc6 krb5kdc[15206](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.6.234.193@KRBTEST.COM for oryx/127.6.234.254@KRBTEST.COM, Server not found in Kerberos database
I20260620 11:43:09.674877 16236 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 11:43:09.665316 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.234.254:38223 (local address 127.6.234.193:55599)
0620 11:43:09.665496 (+ 180us) negotiation.cc:107] Waiting for socket to connect
0620 11:43:09.665513 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0620 11:43:09.665584 (+ 71us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 11:43:09.665813 (+ 229us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 11:43:09.665814 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 11:43:09.665976 (+ 162us) client_negotiation.cc:189] Negotiated authn=SASL
0620 11:43:09.666023 (+ 47us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 11:43:09.666025 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 11:43:09.666728 (+ 703us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 11:43:09.666729 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 11:43:09.667281 (+ 552us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 11:43:09.667283 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 11:43:09.667905 (+ 622us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 11:43:09.667907 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 11:43:09.667978 (+ 71us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 11:43:09.672667 (+ 4689us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 11:43:09.672672 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0620 11:43:09.674778 (+ 2106us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.6.234.254:38223: Server oryx/127.6.234.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":156,"thread_start_us":74,"threads_started":1}
W20260620 11:43:09.674935 16237 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 11:43:09.665482 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.6.234.193:55599 (local address 127.6.234.254:38223)
0620 11:43:09.665608 (+ 126us) server_negotiation.cc:206] Beginning negotiation
0620 11:43:09.665609 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0620 11:43:09.665630 (+ 21us) server_negotiation.cc:407] Connection header received
0620 11:43:09.665651 (+ 21us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 11:43:09.665652 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 11:43:09.665662 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 11:43:09.665722 (+ 60us) server_negotiation.cc:226] Negotiated authn=SASL
0620 11:43:09.666095 (+ 373us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 11:43:09.666638 (+ 543us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 11:43:09.667352 (+ 714us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 11:43:09.667542 (+ 190us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 11:43:09.667628 (+ 86us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 11:43:09.674862 (+ 7234us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.6.234.193:55599: BlockingRecv error: recv got EOF from 127.6.234.193:55599 (error 108)
Metrics: {"server-negotiator.queue_time_us":107,"thread_start_us":80,"threads_started":1}
Jun 20 11:43:09 dist-test-slave-cjc6 krb5kdc[15206](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.6.234.194@KRBTEST.COM for oryx/127.6.234.254@KRBTEST.COM, Server not found in Kerberos database
Jun 20 11:43:09 dist-test-slave-cjc6 krb5kdc[15206](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.6.234.194@KRBTEST.COM for oryx/127.6.234.254@KRBTEST.COM, Server not found in Kerberos database
I20260620 11:43:09.795579 16238 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 11:43:09.790477 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.234.254:38223 (local address 127.6.234.194:44823)
0620 11:43:09.790593 (+ 116us) negotiation.cc:107] Waiting for socket to connect
0620 11:43:09.790604 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0620 11:43:09.790658 (+ 54us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 11:43:09.790798 (+ 140us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 11:43:09.790799 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 11:43:09.790968 (+ 169us) client_negotiation.cc:189] Negotiated authn=SASL
0620 11:43:09.791019 (+ 51us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 11:43:09.791021 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 11:43:09.791922 (+ 901us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 11:43:09.791923 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 11:43:09.792490 (+ 567us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 11:43:09.792492 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 11:43:09.792862 (+ 370us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 11:43:09.792862 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 11:43:09.792902 (+ 40us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 11:43:09.793376 (+ 474us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 11:43:09.793380 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0620 11:43:09.795464 (+ 2084us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.6.234.254:38223: Server oryx/127.6.234.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":94,"thread_start_us":64,"threads_started":1}
W20260620 11:43:09.795616 16237 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 11:43:09.790551 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.6.234.194:44823 (local address 127.6.234.254:38223)
0620 11:43:09.790656 (+ 105us) server_negotiation.cc:206] Beginning negotiation
0620 11:43:09.790659 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0620 11:43:09.790706 (+ 47us) server_negotiation.cc:407] Connection header received
0620 11:43:09.790729 (+ 23us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 11:43:09.790730 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 11:43:09.790743 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 11:43:09.790786 (+ 43us) server_negotiation.cc:226] Negotiated authn=SASL
0620 11:43:09.791098 (+ 312us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 11:43:09.791844 (+ 746us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 11:43:09.792593 (+ 749us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 11:43:09.792785 (+ 192us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 11:43:09.792848 (+ 63us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 11:43:09.795551 (+ 2703us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.6.234.194:44823: BlockingRecv error: recv got EOF from 127.6.234.194:44823 (error 108)
Metrics: {"server-negotiator.queue_time_us":76}
Jun 20 11:43:09 dist-test-slave-cjc6 krb5kdc[15206](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.6.234.195@KRBTEST.COM for oryx/127.6.234.254@KRBTEST.COM, Server not found in Kerberos database
Jun 20 11:43:09 dist-test-slave-cjc6 krb5kdc[15206](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.6.234.195@KRBTEST.COM for oryx/127.6.234.254@KRBTEST.COM, Server not found in Kerberos database
I20260620 11:43:09.916505 16239 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 11:43:09.911305 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.234.254:38223 (local address 127.6.234.195:52803)
0620 11:43:09.911454 (+ 149us) negotiation.cc:107] Waiting for socket to connect
0620 11:43:09.911499 (+ 45us) client_negotiation.cc:174] Beginning negotiation
0620 11:43:09.911572 (+ 73us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 11:43:09.911752 (+ 180us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 11:43:09.911753 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 11:43:09.911933 (+ 180us) client_negotiation.cc:189] Negotiated authn=SASL
0620 11:43:09.911979 (+ 46us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 11:43:09.911981 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 11:43:09.912764 (+ 783us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 11:43:09.912765 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 11:43:09.913405 (+ 640us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 11:43:09.913408 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 11:43:09.913911 (+ 503us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 11:43:09.913912 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 11:43:09.913971 (+ 59us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 11:43:09.914557 (+ 586us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 11:43:09.914561 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0620 11:43:09.916411 (+ 1850us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.6.234.254:38223: Server oryx/127.6.234.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":131,"thread_start_us":100,"threads_started":1}
W20260620 11:43:09.916556 16237 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 11:43:09.911440 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.6.234.195:52803 (local address 127.6.234.254:38223)
0620 11:43:09.911533 (+ 93us) server_negotiation.cc:206] Beginning negotiation
0620 11:43:09.911535 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 11:43:09.911550 (+ 15us) server_negotiation.cc:407] Connection header received
0620 11:43:09.911656 (+ 106us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 11:43:09.911657 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 11:43:09.911673 (+ 16us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 11:43:09.911727 (+ 54us) server_negotiation.cc:226] Negotiated authn=SASL
0620 11:43:09.912068 (+ 341us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 11:43:09.912677 (+ 609us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 11:43:09.913502 (+ 825us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 11:43:09.913808 (+ 306us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 11:43:09.913876 (+ 68us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 11:43:09.916490 (+ 2614us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.6.234.195:52803: BlockingRecv error: recv got EOF from 127.6.234.195:52803 (error 108)
Metrics: {"server-negotiator.queue_time_us":68}
Jun 20 11:43:10 dist-test-slave-cjc6 krb5kdc[15206](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.6.234.193@KRBTEST.COM for oryx/127.6.234.254@KRBTEST.COM, Server not found in Kerberos database
Jun 20 11:43:10 dist-test-slave-cjc6 krb5kdc[15206](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.6.234.193@KRBTEST.COM for oryx/127.6.234.254@KRBTEST.COM, Server not found in Kerberos database
I20260620 11:43:10.680042 16240 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 11:43:10.675377 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.234.254:38223 (local address 127.6.234.193:60375)
0620 11:43:10.675520 (+ 143us) negotiation.cc:107] Waiting for socket to connect
0620 11:43:10.675533 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0620 11:43:10.675586 (+ 53us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 11:43:10.675747 (+ 161us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 11:43:10.675748 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 11:43:10.675905 (+ 157us) client_negotiation.cc:189] Negotiated authn=SASL
0620 11:43:10.675953 (+ 48us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 11:43:10.675954 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 11:43:10.676702 (+ 748us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 11:43:10.676703 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 11:43:10.677220 (+ 517us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 11:43:10.677221 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 11:43:10.677580 (+ 359us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 11:43:10.677581 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 11:43:10.677618 (+ 37us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 11:43:10.678123 (+ 505us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 11:43:10.678127 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0620 11:43:10.679938 (+ 1811us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.6.234.254:38223: Server oryx/127.6.234.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":122,"thread_start_us":86,"threads_started":1}
W20260620 11:43:10.680101 16241 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 11:43:10.675479 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.6.234.193:60375 (local address 127.6.234.254:38223)
0620 11:43:10.675609 (+ 130us) server_negotiation.cc:206] Beginning negotiation
0620 11:43:10.675612 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0620 11:43:10.675621 (+ 9us) server_negotiation.cc:407] Connection header received
0620 11:43:10.675651 (+ 30us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 11:43:10.675652 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 11:43:10.675662 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 11:43:10.675732 (+ 70us) server_negotiation.cc:226] Negotiated authn=SASL
0620 11:43:10.676055 (+ 323us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 11:43:10.676622 (+ 567us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 11:43:10.677312 (+ 690us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 11:43:10.677510 (+ 198us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 11:43:10.677571 (+ 61us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 11:43:10.680018 (+ 2447us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.6.234.193:60375: BlockingRecv error: recv got EOF from 127.6.234.193:60375 (error 108)
Metrics: {"server-negotiator.queue_time_us":107,"thread_start_us":114,"threads_started":1}
Jun 20 11:43:10 dist-test-slave-cjc6 krb5kdc[15206](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.6.234.194@KRBTEST.COM for oryx/127.6.234.254@KRBTEST.COM, Server not found in Kerberos database
Jun 20 11:43:10 dist-test-slave-cjc6 krb5kdc[15206](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.6.234.194@KRBTEST.COM for oryx/127.6.234.254@KRBTEST.COM, Server not found in Kerberos database
I20260620 11:43:10.801062 16242 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 11:43:10.796178 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.234.254:38223 (local address 127.6.234.194:37077)
0620 11:43:10.796369 (+ 191us) negotiation.cc:107] Waiting for socket to connect
0620 11:43:10.796381 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0620 11:43:10.796445 (+ 64us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 11:43:10.796618 (+ 173us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 11:43:10.796619 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 11:43:10.796792 (+ 173us) client_negotiation.cc:189] Negotiated authn=SASL
0620 11:43:10.796847 (+ 55us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 11:43:10.796848 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 11:43:10.797515 (+ 667us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 11:43:10.797517 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 11:43:10.798168 (+ 651us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 11:43:10.798170 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 11:43:10.798527 (+ 357us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 11:43:10.798528 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 11:43:10.798590 (+ 62us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 11:43:10.799046 (+ 456us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 11:43:10.799051 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0620 11:43:10.801005 (+ 1954us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.6.234.254:38223: Server oryx/127.6.234.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":164,"thread_start_us":120,"threads_started":1}
W20260620 11:43:10.801226 16241 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 11:43:10.796289 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.6.234.194:37077 (local address 127.6.234.254:38223)
0620 11:43:10.796374 (+ 85us) server_negotiation.cc:206] Beginning negotiation
0620 11:43:10.796376 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 11:43:10.796444 (+ 68us) server_negotiation.cc:407] Connection header received
0620 11:43:10.796494 (+ 50us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 11:43:10.796495 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 11:43:10.796507 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 11:43:10.796563 (+ 56us) server_negotiation.cc:226] Negotiated authn=SASL
0620 11:43:10.796914 (+ 351us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 11:43:10.797406 (+ 492us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 11:43:10.798256 (+ 850us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 11:43:10.798453 (+ 197us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 11:43:10.798511 (+ 58us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 11:43:10.801146 (+ 2635us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.6.234.194:37077: BlockingRecv error: recv got EOF from 127.6.234.194:37077 (error 108)
Metrics: {"server-negotiator.queue_time_us":52}
Jun 20 11:43:10 dist-test-slave-cjc6 krb5kdc[15206](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.6.234.195@KRBTEST.COM for oryx/127.6.234.254@KRBTEST.COM, Server not found in Kerberos database
Jun 20 11:43:10 dist-test-slave-cjc6 krb5kdc[15206](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.6.234.195@KRBTEST.COM for oryx/127.6.234.254@KRBTEST.COM, Server not found in Kerberos database
I20260620 11:43:10.921897 16243 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 11:43:10.917007 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.234.254:38223 (local address 127.6.234.195:39413)
0620 11:43:10.917141 (+ 134us) negotiation.cc:107] Waiting for socket to connect
0620 11:43:10.917153 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0620 11:43:10.917221 (+ 68us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 11:43:10.917381 (+ 160us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 11:43:10.917382 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 11:43:10.917565 (+ 183us) client_negotiation.cc:189] Negotiated authn=SASL
0620 11:43:10.917612 (+ 47us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 11:43:10.917614 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 11:43:10.918300 (+ 686us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 11:43:10.918301 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 11:43:10.918851 (+ 550us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 11:43:10.918853 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 11:43:10.919229 (+ 376us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 11:43:10.919230 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 11:43:10.919291 (+ 61us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 11:43:10.919730 (+ 439us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 11:43:10.919735 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0620 11:43:10.921829 (+ 2094us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.6.234.254:38223: Server oryx/127.6.234.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":114,"thread_start_us":87,"threads_started":1}
W20260620 11:43:10.921949 16241 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 11:43:10.917139 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.6.234.195:39413 (local address 127.6.234.254:38223)
0620 11:43:10.917195 (+ 56us) server_negotiation.cc:206] Beginning negotiation
0620 11:43:10.917197 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 11:43:10.917207 (+ 10us) server_negotiation.cc:407] Connection header received
0620 11:43:10.917287 (+ 80us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 11:43:10.917288 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 11:43:10.917303 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 11:43:10.917356 (+ 53us) server_negotiation.cc:226] Negotiated authn=SASL
0620 11:43:10.917718 (+ 362us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 11:43:10.918207 (+ 489us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 11:43:10.918942 (+ 735us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 11:43:10.919140 (+ 198us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 11:43:10.919227 (+ 87us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 11:43:10.921903 (+ 2676us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.6.234.195:39413: BlockingRecv error: recv got EOF from 127.6.234.195:39413 (error 108)
Metrics: {"server-negotiator.queue_time_us":30}
Jun 20 11:43:11 dist-test-slave-cjc6 krb5kdc[15206](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.6.234.193@KRBTEST.COM for oryx/127.6.234.254@KRBTEST.COM, Server not found in Kerberos database
Jun 20 11:43:11 dist-test-slave-cjc6 krb5kdc[15206](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.6.234.193@KRBTEST.COM for oryx/127.6.234.254@KRBTEST.COM, Server not found in Kerberos database
I20260620 11:43:11.685124 16244 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 11:43:11.680539 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.234.254:38223 (local address 127.6.234.193:43409)
0620 11:43:11.680696 (+ 157us) negotiation.cc:107] Waiting for socket to connect
0620 11:43:11.680709 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0620 11:43:11.680786 (+ 77us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 11:43:11.680930 (+ 144us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 11:43:11.680931 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 11:43:11.681095 (+ 164us) client_negotiation.cc:189] Negotiated authn=SASL
0620 11:43:11.681144 (+ 49us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 11:43:11.681145 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 11:43:11.681828 (+ 683us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 11:43:11.681829 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 11:43:11.682343 (+ 514us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 11:43:11.682344 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 11:43:11.682685 (+ 341us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 11:43:11.682687 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 11:43:11.682730 (+ 43us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 11:43:11.683267 (+ 537us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 11:43:11.683272 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0620 11:43:11.685031 (+ 1759us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.6.234.254:38223: Server oryx/127.6.234.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":132,"thread_start_us":70,"threads_started":1}
W20260620 11:43:11.685173 16245 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 11:43:11.680590 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.6.234.193:43409 (local address 127.6.234.254:38223)
0620 11:43:11.680734 (+ 144us) server_negotiation.cc:206] Beginning negotiation
0620 11:43:11.680736 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 11:43:11.680757 (+ 21us) server_negotiation.cc:407] Connection header received
0620 11:43:11.680843 (+ 86us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 11:43:11.680844 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 11:43:11.680854 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 11:43:11.680911 (+ 57us) server_negotiation.cc:226] Negotiated authn=SASL
0620 11:43:11.681204 (+ 293us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 11:43:11.681757 (+ 553us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 11:43:11.682427 (+ 670us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 11:43:11.682614 (+ 187us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 11:43:11.682665 (+ 51us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 11:43:11.685103 (+ 2438us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.6.234.193:43409: BlockingRecv error: recv got EOF from 127.6.234.193:43409 (error 108)
Metrics: {"server-negotiator.queue_time_us":126,"thread_start_us":85,"threads_started":1}
Jun 20 11:43:11 dist-test-slave-cjc6 krb5kdc[15206](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.6.234.194@KRBTEST.COM for oryx/127.6.234.254@KRBTEST.COM, Server not found in Kerberos database
Jun 20 11:43:11 dist-test-slave-cjc6 krb5kdc[15206](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.6.234.194@KRBTEST.COM for oryx/127.6.234.254@KRBTEST.COM, Server not found in Kerberos database
I20260620 11:43:11.806702 16246 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 11:43:11.801550 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.234.254:38223 (local address 127.6.234.194:39819)
0620 11:43:11.801750 (+ 200us) negotiation.cc:107] Waiting for socket to connect
0620 11:43:11.801772 (+ 22us) client_negotiation.cc:174] Beginning negotiation
0620 11:43:11.801859 (+ 87us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 11:43:11.801992 (+ 133us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 11:43:11.801992 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 11:43:11.802174 (+ 182us) client_negotiation.cc:189] Negotiated authn=SASL
0620 11:43:11.802220 (+ 46us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 11:43:11.802221 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 11:43:11.802904 (+ 683us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 11:43:11.802905 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 11:43:11.803725 (+ 820us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 11:43:11.803728 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 11:43:11.804189 (+ 461us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 11:43:11.804190 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 11:43:11.804250 (+ 60us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 11:43:11.804686 (+ 436us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 11:43:11.804691 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0620 11:43:11.806640 (+ 1949us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.6.234.254:38223: Server oryx/127.6.234.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":182,"thread_start_us":100,"threads_started":1}
W20260620 11:43:11.806839 16245 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 11:43:11.801639 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.6.234.194:39819 (local address 127.6.234.254:38223)
0620 11:43:11.801735 (+ 96us) server_negotiation.cc:206] Beginning negotiation
0620 11:43:11.801738 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0620 11:43:11.801841 (+ 103us) server_negotiation.cc:407] Connection header received
0620 11:43:11.801919 (+ 78us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 11:43:11.801920 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 11:43:11.801933 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 11:43:11.801977 (+ 44us) server_negotiation.cc:226] Negotiated authn=SASL
0620 11:43:11.802306 (+ 329us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 11:43:11.802805 (+ 499us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 11:43:11.803848 (+ 1043us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 11:43:11.804089 (+ 241us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 11:43:11.804192 (+ 103us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 11:43:11.806746 (+ 2554us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.6.234.194:39819: BlockingRecv error: recv got EOF from 127.6.234.194:39819 (error 108)
Metrics: {"server-negotiator.queue_time_us":46}
Jun 20 11:43:11 dist-test-slave-cjc6 krb5kdc[15206](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.6.234.195@KRBTEST.COM for oryx/127.6.234.254@KRBTEST.COM, Server not found in Kerberos database
Jun 20 11:43:11 dist-test-slave-cjc6 krb5kdc[15206](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.6.234.195@KRBTEST.COM for oryx/127.6.234.254@KRBTEST.COM, Server not found in Kerberos database
I20260620 11:43:11.927098 16247 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 11:43:11.922388 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.234.254:38223 (local address 127.6.234.195:43671)
0620 11:43:11.922539 (+ 151us) negotiation.cc:107] Waiting for socket to connect
0620 11:43:11.922550 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0620 11:43:11.922623 (+ 73us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 11:43:11.922758 (+ 135us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 11:43:11.922758 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 11:43:11.922936 (+ 178us) client_negotiation.cc:189] Negotiated authn=SASL
0620 11:43:11.922984 (+ 48us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 11:43:11.922986 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 11:43:11.923673 (+ 687us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 11:43:11.923675 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 11:43:11.924201 (+ 526us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 11:43:11.924203 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 11:43:11.924611 (+ 408us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 11:43:11.924612 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 11:43:11.924678 (+ 66us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 11:43:11.925139 (+ 461us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 11:43:11.925143 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0620 11:43:11.927036 (+ 1893us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.6.234.254:38223: Server oryx/127.6.234.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":129,"thread_start_us":90,"threads_started":1}
W20260620 11:43:11.927280 16245 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 11:43:11.922554 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.6.234.195:43671 (local address 127.6.234.254:38223)
0620 11:43:11.922625 (+ 71us) server_negotiation.cc:206] Beginning negotiation
0620 11:43:11.922628 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0620 11:43:11.922636 (+ 8us) server_negotiation.cc:407] Connection header received
0620 11:43:11.922686 (+ 50us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 11:43:11.922687 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 11:43:11.922702 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 11:43:11.922753 (+ 51us) server_negotiation.cc:226] Negotiated authn=SASL
0620 11:43:11.923067 (+ 314us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 11:43:11.923587 (+ 520us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 11:43:11.924329 (+ 742us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 11:43:11.924521 (+ 192us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 11:43:11.924608 (+ 87us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 11:43:11.927198 (+ 2590us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.6.234.195:43671: BlockingRecv error: recv got EOF from 127.6.234.195:43671 (error 108)
Metrics: {"server-negotiator.queue_time_us":46}
Jun 20 11:43:12 dist-test-slave-cjc6 krb5kdc[15206](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.6.234.193@KRBTEST.COM for oryx/127.6.234.254@KRBTEST.COM, Server not found in Kerberos database
Jun 20 11:43:12 dist-test-slave-cjc6 krb5kdc[15206](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.6.234.193@KRBTEST.COM for oryx/127.6.234.254@KRBTEST.COM, Server not found in Kerberos database
I20260620 11:43:12.690207 16248 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 11:43:12.685623 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.234.254:38223 (local address 127.6.234.193:42331)
0620 11:43:12.685777 (+ 154us) negotiation.cc:107] Waiting for socket to connect
0620 11:43:12.685791 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0620 11:43:12.685859 (+ 68us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 11:43:12.685981 (+ 122us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 11:43:12.685982 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 11:43:12.686144 (+ 162us) client_negotiation.cc:189] Negotiated authn=SASL
0620 11:43:12.686189 (+ 45us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 11:43:12.686191 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 11:43:12.686849 (+ 658us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 11:43:12.686850 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 11:43:12.687392 (+ 542us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 11:43:12.687393 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 11:43:12.687740 (+ 347us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 11:43:12.687740 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 11:43:12.687780 (+ 40us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 11:43:12.688204 (+ 424us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 11:43:12.688208 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0620 11:43:12.690100 (+ 1892us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.6.234.254:38223: Server oryx/127.6.234.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":131,"thread_start_us":110,"threads_started":1}
W20260620 11:43:12.690296 16249 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 11:43:12.685676 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.6.234.193:42331 (local address 127.6.234.254:38223)
0620 11:43:12.685844 (+ 168us) server_negotiation.cc:206] Beginning negotiation
0620 11:43:12.685846 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 11:43:12.685853 (+ 7us) server_negotiation.cc:407] Connection header received
0620 11:43:12.685909 (+ 56us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 11:43:12.685909 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 11:43:12.685920 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 11:43:12.685970 (+ 50us) server_negotiation.cc:226] Negotiated authn=SASL
0620 11:43:12.686267 (+ 297us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 11:43:12.686775 (+ 508us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 11:43:12.687467 (+ 692us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 11:43:12.687670 (+ 203us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 11:43:12.687725 (+ 55us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 11:43:12.690207 (+ 2482us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.6.234.193:42331: BlockingRecv error: recv got EOF from 127.6.234.193:42331 (error 108)
Metrics: {"server-negotiator.queue_time_us":148,"thread_start_us":114,"threads_started":1}
Jun 20 11:43:12 dist-test-slave-cjc6 krb5kdc[15206](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.6.234.194@KRBTEST.COM for oryx/127.6.234.254@KRBTEST.COM, Server not found in Kerberos database
Jun 20 11:43:12 dist-test-slave-cjc6 krb5kdc[15206](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.6.234.194@KRBTEST.COM for oryx/127.6.234.254@KRBTEST.COM, Server not found in Kerberos database
I20260620 11:43:12.811932 16250 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 11:43:12.807242 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.234.254:38223 (local address 127.6.234.194:43263)
0620 11:43:12.807404 (+ 162us) negotiation.cc:107] Waiting for socket to connect
0620 11:43:12.807416 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0620 11:43:12.807501 (+ 85us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 11:43:12.807684 (+ 183us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 11:43:12.807685 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 11:43:12.807876 (+ 191us) client_negotiation.cc:189] Negotiated authn=SASL
0620 11:43:12.807921 (+ 45us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 11:43:12.807923 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 11:43:12.808590 (+ 667us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 11:43:12.808592 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 11:43:12.809143 (+ 551us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 11:43:12.809145 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 11:43:12.809522 (+ 377us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 11:43:12.809523 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 11:43:12.809582 (+ 59us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 11:43:12.810098 (+ 516us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 11:43:12.810109 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0620 11:43:12.811835 (+ 1726us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.6.234.254:38223: Server oryx/127.6.234.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":141,"thread_start_us":90,"threads_started":1}
W20260620 11:43:12.811978 16249 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 11:43:12.807437 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.6.234.194:43263 (local address 127.6.234.254:38223)
0620 11:43:12.807516 (+ 79us) server_negotiation.cc:206] Beginning negotiation
0620 11:43:12.807518 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 11:43:12.807526 (+ 8us) server_negotiation.cc:407] Connection header received
0620 11:43:12.807557 (+ 31us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 11:43:12.807558 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 11:43:12.807570 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 11:43:12.807672 (+ 102us) server_negotiation.cc:226] Negotiated authn=SASL
0620 11:43:12.808002 (+ 330us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 11:43:12.808501 (+ 499us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 11:43:12.809232 (+ 731us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 11:43:12.809433 (+ 201us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 11:43:12.809519 (+ 86us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 11:43:12.811914 (+ 2395us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.6.234.194:43263: BlockingRecv error: recv got EOF from 127.6.234.194:43263 (error 108)
Metrics: {"server-negotiator.queue_time_us":51}
Jun 20 11:43:12 dist-test-slave-cjc6 krb5kdc[15206](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.6.234.195@KRBTEST.COM for oryx/127.6.234.254@KRBTEST.COM, Server not found in Kerberos database
Jun 20 11:43:12 dist-test-slave-cjc6 krb5kdc[15206](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.6.234.195@KRBTEST.COM for oryx/127.6.234.254@KRBTEST.COM, Server not found in Kerberos database
I20260620 11:43:12.932842 16251 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 11:43:12.927652 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.234.254:38223 (local address 127.6.234.195:45587)
0620 11:43:12.927791 (+ 139us) negotiation.cc:107] Waiting for socket to connect
0620 11:43:12.927803 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0620 11:43:12.927867 (+ 64us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 11:43:12.928000 (+ 133us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 11:43:12.928000 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 11:43:12.928182 (+ 182us) client_negotiation.cc:189] Negotiated authn=SASL
0620 11:43:12.928256 (+ 74us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 11:43:12.928258 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 11:43:12.928993 (+ 735us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 11:43:12.928994 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 11:43:12.929603 (+ 609us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 11:43:12.929605 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 11:43:12.930076 (+ 471us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 11:43:12.930077 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 11:43:12.930120 (+ 43us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 11:43:12.930696 (+ 576us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 11:43:12.930701 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0620 11:43:12.932746 (+ 2045us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.6.234.254:38223: Server oryx/127.6.234.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":121,"thread_start_us":89,"threads_started":1}
W20260620 11:43:12.932943 16249 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 11:43:12.927715 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.6.234.195:45587 (local address 127.6.234.254:38223)
0620 11:43:12.927780 (+ 65us) server_negotiation.cc:206] Beginning negotiation
0620 11:43:12.927783 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0620 11:43:12.927869 (+ 86us) server_negotiation.cc:407] Connection header received
0620 11:43:12.927922 (+ 53us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 11:43:12.927922 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 11:43:12.927936 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 11:43:12.927977 (+ 41us) server_negotiation.cc:226] Negotiated authn=SASL
0620 11:43:12.928353 (+ 376us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 11:43:12.928884 (+ 531us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 11:43:12.929761 (+ 877us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 11:43:12.930003 (+ 242us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 11:43:12.930059 (+ 56us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 11:43:12.932873 (+ 2814us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.6.234.195:45587: BlockingRecv error: recv got EOF from 127.6.234.195:45587 (error 108)
Metrics: {"server-negotiator.queue_time_us":40}
Jun 20 11:43:13 dist-test-slave-cjc6 krb5kdc[15206](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.6.234.193@KRBTEST.COM for oryx/127.6.234.254@KRBTEST.COM, Server not found in Kerberos database
Jun 20 11:43:13 dist-test-slave-cjc6 krb5kdc[15206](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.6.234.193@KRBTEST.COM for oryx/127.6.234.254@KRBTEST.COM, Server not found in Kerberos database
I20260620 11:43:13.695344 16252 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 11:43:13.690664 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.234.254:38223 (local address 127.6.234.193:38667)
0620 11:43:13.690839 (+ 175us) negotiation.cc:107] Waiting for socket to connect
0620 11:43:13.690852 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0620 11:43:13.690927 (+ 75us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 11:43:13.691064 (+ 137us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 11:43:13.691064 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 11:43:13.691236 (+ 172us) client_negotiation.cc:189] Negotiated authn=SASL
0620 11:43:13.691282 (+ 46us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 11:43:13.691283 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 11:43:13.691912 (+ 629us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 11:43:13.691913 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 11:43:13.692629 (+ 716us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 11:43:13.692632 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 11:43:13.692988 (+ 356us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 11:43:13.692989 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 11:43:13.693034 (+ 45us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 11:43:13.693465 (+ 431us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 11:43:13.693469 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0620 11:43:13.695276 (+ 1807us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.6.234.254:38223: Server oryx/127.6.234.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":151,"thread_start_us":79,"threads_started":1}
W20260620 11:43:13.695487 16253 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 11:43:13.690791 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.6.234.193:38667 (local address 127.6.234.254:38223)
0620 11:43:13.690922 (+ 131us) server_negotiation.cc:206] Beginning negotiation
0620 11:43:13.690924 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 11:43:13.690933 (+ 9us) server_negotiation.cc:407] Connection header received
0620 11:43:13.690982 (+ 49us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 11:43:13.690983 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 11:43:13.690994 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 11:43:13.691044 (+ 50us) server_negotiation.cc:226] Negotiated authn=SASL
0620 11:43:13.691346 (+ 302us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 11:43:13.691842 (+ 496us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 11:43:13.692703 (+ 861us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 11:43:13.692915 (+ 212us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 11:43:13.692970 (+ 55us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 11:43:13.695394 (+ 2424us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.6.234.193:38667: BlockingRecv error: recv got EOF from 127.6.234.193:38667 (error 108)
Metrics: {"server-negotiator.queue_time_us":111,"thread_start_us":63,"threads_started":1}
Jun 20 11:43:13 dist-test-slave-cjc6 krb5kdc[15206](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.6.234.194@KRBTEST.COM for oryx/127.6.234.254@KRBTEST.COM, Server not found in Kerberos database
Jun 20 11:43:13 dist-test-slave-cjc6 krb5kdc[15206](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.6.234.194@KRBTEST.COM for oryx/127.6.234.254@KRBTEST.COM, Server not found in Kerberos database
I20260620 11:43:13.817294 16254 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 11:43:13.812428 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.234.254:38223 (local address 127.6.234.194:44863)
0620 11:43:13.812572 (+ 144us) negotiation.cc:107] Waiting for socket to connect
0620 11:43:13.812584 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0620 11:43:13.812643 (+ 59us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 11:43:13.812797 (+ 154us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 11:43:13.812798 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 11:43:13.812966 (+ 168us) client_negotiation.cc:189] Negotiated authn=SASL
0620 11:43:13.813012 (+ 46us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 11:43:13.813014 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 11:43:13.813807 (+ 793us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 11:43:13.813808 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 11:43:13.814339 (+ 531us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 11:43:13.814341 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 11:43:13.814680 (+ 339us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 11:43:13.814681 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 11:43:13.814735 (+ 54us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 11:43:13.815168 (+ 433us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 11:43:13.815172 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0620 11:43:13.817237 (+ 2065us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.6.234.254:38223: Server oryx/127.6.234.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":123,"thread_start_us":83,"threads_started":1}
W20260620 11:43:13.817476 16253 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 11:43:13.812585 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.6.234.194:44863 (local address 127.6.234.254:38223)
0620 11:43:13.812654 (+ 69us) server_negotiation.cc:206] Beginning negotiation
0620 11:43:13.812656 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 11:43:13.812669 (+ 13us) server_negotiation.cc:407] Connection header received
0620 11:43:13.812695 (+ 26us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 11:43:13.812695 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 11:43:13.812707 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 11:43:13.812776 (+ 69us) server_negotiation.cc:226] Negotiated authn=SASL
0620 11:43:13.813095 (+ 319us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 11:43:13.813736 (+ 641us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 11:43:13.814407 (+ 671us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 11:43:13.814590 (+ 183us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 11:43:13.814665 (+ 75us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 11:43:13.817402 (+ 2737us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.6.234.194:44863: BlockingRecv error: recv got EOF from 127.6.234.194:44863 (error 108)
Metrics: {"server-negotiator.queue_time_us":42}
Jun 20 11:43:13 dist-test-slave-cjc6 krb5kdc[15206](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.6.234.195@KRBTEST.COM for oryx/127.6.234.254@KRBTEST.COM, Server not found in Kerberos database
Jun 20 11:43:13 dist-test-slave-cjc6 krb5kdc[15206](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.6.234.195@KRBTEST.COM for oryx/127.6.234.254@KRBTEST.COM, Server not found in Kerberos database
I20260620 11:43:13.938015 16255 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 11:43:13.933324 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.234.254:38223 (local address 127.6.234.195:36593)
0620 11:43:13.933446 (+ 122us) negotiation.cc:107] Waiting for socket to connect
0620 11:43:13.933460 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0620 11:43:13.933513 (+ 53us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 11:43:13.933681 (+ 168us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 11:43:13.933682 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 11:43:13.933860 (+ 178us) client_negotiation.cc:189] Negotiated authn=SASL
0620 11:43:13.933909 (+ 49us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 11:43:13.933911 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 11:43:13.934729 (+ 818us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 11:43:13.934730 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 11:43:13.935292 (+ 562us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 11:43:13.935294 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 11:43:13.935602 (+ 308us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 11:43:13.935603 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 11:43:13.935654 (+ 51us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 11:43:13.936158 (+ 504us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 11:43:13.936164 (+ 6us) client_negotiation.cc:656] calling sasl_client_start()
0620 11:43:13.937949 (+ 1785us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.6.234.254:38223: Server oryx/127.6.234.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":100,"thread_start_us":70,"threads_started":1}
W20260620 11:43:13.938133 16253 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 11:43:13.933406 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.6.234.195:36593 (local address 127.6.234.254:38223)
0620 11:43:13.933487 (+ 81us) server_negotiation.cc:206] Beginning negotiation
0620 11:43:13.933494 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0620 11:43:13.933505 (+ 11us) server_negotiation.cc:407] Connection header received
0620 11:43:13.933591 (+ 86us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 11:43:13.933592 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 11:43:13.933606 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 11:43:13.933660 (+ 54us) server_negotiation.cc:226] Negotiated authn=SASL
0620 11:43:13.933970 (+ 310us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 11:43:13.934673 (+ 703us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 11:43:13.935357 (+ 684us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 11:43:13.935530 (+ 173us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 11:43:13.935577 (+ 47us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 11:43:13.938036 (+ 2459us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.6.234.195:36593: BlockingRecv error: recv got EOF from 127.6.234.195:36593 (error 108)
Metrics: {"server-negotiator.queue_time_us":54}
Jun 20 11:43:14 dist-test-slave-cjc6 krb5kdc[15206](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.6.234.193@KRBTEST.COM for oryx/127.6.234.254@KRBTEST.COM, Server not found in Kerberos database
Jun 20 11:43:14 dist-test-slave-cjc6 krb5kdc[15206](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.6.234.193@KRBTEST.COM for oryx/127.6.234.254@KRBTEST.COM, Server not found in Kerberos database
I20260620 11:43:14.705101 16256 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 11:43:14.695844 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.234.254:38223 (local address 127.6.234.193:40657)
0620 11:43:14.696016 (+ 172us) negotiation.cc:107] Waiting for socket to connect
0620 11:43:14.696028 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0620 11:43:14.696085 (+ 57us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 11:43:14.696224 (+ 139us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 11:43:14.696224 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 11:43:14.696397 (+ 173us) client_negotiation.cc:189] Negotiated authn=SASL
0620 11:43:14.696443 (+ 46us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 11:43:14.696444 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 11:43:14.697187 (+ 743us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 11:43:14.697188 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 11:43:14.697799 (+ 611us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 11:43:14.697801 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 11:43:14.698198 (+ 397us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 11:43:14.698199 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 11:43:14.698250 (+ 51us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 11:43:14.703252 (+ 5002us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 11:43:14.703257 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0620 11:43:14.705041 (+ 1784us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.6.234.254:38223: Server oryx/127.6.234.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":151,"thread_start_us":92,"threads_started":1}
W20260620 11:43:14.705302 16257 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 11:43:14.695949 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.6.234.193:40657 (local address 127.6.234.254:38223)
0620 11:43:14.696063 (+ 114us) server_negotiation.cc:206] Beginning negotiation
0620 11:43:14.696064 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0620 11:43:14.696073 (+ 9us) server_negotiation.cc:407] Connection header received
0620 11:43:14.696141 (+ 68us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 11:43:14.696142 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 11:43:14.696153 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 11:43:14.696205 (+ 52us) server_negotiation.cc:226] Negotiated authn=SASL
0620 11:43:14.696549 (+ 344us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 11:43:14.697061 (+ 512us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 11:43:14.697900 (+ 839us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 11:43:14.698103 (+ 203us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 11:43:14.698168 (+ 65us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 11:43:14.705203 (+ 7035us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.6.234.193:40657: BlockingRecv error: recv got EOF from 127.6.234.193:40657 (error 108)
Metrics: {"server-negotiator.queue_time_us":86,"thread_start_us":63,"threads_started":1}
Jun 20 11:43:14 dist-test-slave-cjc6 krb5kdc[15206](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.6.234.194@KRBTEST.COM for oryx/127.6.234.254@KRBTEST.COM, Server not found in Kerberos database
Jun 20 11:43:14 dist-test-slave-cjc6 krb5kdc[15206](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.6.234.194@KRBTEST.COM for oryx/127.6.234.254@KRBTEST.COM, Server not found in Kerberos database
I20260620 11:43:14.823071 16258 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 11:43:14.817921 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.234.254:38223 (local address 127.6.234.194:48145)
0620 11:43:14.818051 (+ 130us) negotiation.cc:107] Waiting for socket to connect
0620 11:43:14.818066 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0620 11:43:14.818130 (+ 64us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 11:43:14.818278 (+ 148us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 11:43:14.818278 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 11:43:14.818444 (+ 166us) client_negotiation.cc:189] Negotiated authn=SASL
0620 11:43:14.818498 (+ 54us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 11:43:14.818500 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 11:43:14.819508 (+ 1008us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 11:43:14.819509 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 11:43:14.820198 (+ 689us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 11:43:14.820201 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 11:43:14.820598 (+ 397us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 11:43:14.820599 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 11:43:14.820659 (+ 60us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 11:43:14.821115 (+ 456us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 11:43:14.821121 (+ 6us) client_negotiation.cc:656] calling sasl_client_start()
0620 11:43:14.822934 (+ 1813us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.6.234.254:38223: Server oryx/127.6.234.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":102,"thread_start_us":67,"threads_started":1}
W20260620 11:43:14.823091 16257 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 11:43:14.818051 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.6.234.194:48145 (local address 127.6.234.254:38223)
0620 11:43:14.818128 (+ 77us) server_negotiation.cc:206] Beginning negotiation
0620 11:43:14.818130 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 11:43:14.818141 (+ 11us) server_negotiation.cc:407] Connection header received
0620 11:43:14.818184 (+ 43us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 11:43:14.818185 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 11:43:14.818197 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 11:43:14.818255 (+ 58us) server_negotiation.cc:226] Negotiated authn=SASL
0620 11:43:14.818608 (+ 353us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 11:43:14.819416 (+ 808us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 11:43:14.820286 (+ 870us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 11:43:14.820507 (+ 221us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 11:43:14.820586 (+ 79us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 11:43:14.823026 (+ 2440us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.6.234.194:48145: BlockingRecv error: recv got EOF from 127.6.234.194:48145 (error 108)
Metrics: {"server-negotiator.queue_time_us":48}
Jun 20 11:43:14 dist-test-slave-cjc6 krb5kdc[15206](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.6.234.195@KRBTEST.COM for oryx/127.6.234.254@KRBTEST.COM, Server not found in Kerberos database
Jun 20 11:43:14 dist-test-slave-cjc6 krb5kdc[15206](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.6.234.195@KRBTEST.COM for oryx/127.6.234.254@KRBTEST.COM, Server not found in Kerberos database
I20260620 11:43:14.943487 16259 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 11:43:14.938533 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.234.254:38223 (local address 127.6.234.195:52323)
0620 11:43:14.938693 (+ 160us) negotiation.cc:107] Waiting for socket to connect
0620 11:43:14.938706 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0620 11:43:14.938795 (+ 89us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 11:43:14.938961 (+ 166us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 11:43:14.938962 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 11:43:14.939150 (+ 188us) client_negotiation.cc:189] Negotiated authn=SASL
0620 11:43:14.939197 (+ 47us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 11:43:14.939200 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 11:43:14.939948 (+ 748us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 11:43:14.939950 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 11:43:14.940478 (+ 528us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 11:43:14.940480 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 11:43:14.940883 (+ 403us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 11:43:14.940884 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 11:43:14.940947 (+ 63us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 11:43:14.941497 (+ 550us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 11:43:14.941502 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0620 11:43:14.943390 (+ 1888us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.6.234.254:38223: Server oryx/127.6.234.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":140,"thread_start_us":112,"threads_started":1}
W20260620 11:43:14.943535 16257 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 11:43:14.938582 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.6.234.195:52323 (local address 127.6.234.254:38223)
0620 11:43:14.938660 (+ 78us) server_negotiation.cc:206] Beginning negotiation
0620 11:43:14.938662 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 11:43:14.938779 (+ 117us) server_negotiation.cc:407] Connection header received
0620 11:43:14.938861 (+ 82us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 11:43:14.938862 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 11:43:14.938878 (+ 16us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 11:43:14.938953 (+ 75us) server_negotiation.cc:226] Negotiated authn=SASL
0620 11:43:14.939281 (+ 328us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 11:43:14.939858 (+ 577us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 11:43:14.940587 (+ 729us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 11:43:14.940816 (+ 229us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 11:43:14.940871 (+ 55us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 11:43:14.943470 (+ 2599us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.6.234.195:52323: BlockingRecv error: recv got EOF from 127.6.234.195:52323 (error 108)
Metrics: {"server-negotiator.queue_time_us":50}
Jun 20 11:43:15 dist-test-slave-cjc6 krb5kdc[15206](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.6.234.193@KRBTEST.COM for oryx/127.6.234.254@KRBTEST.COM, Server not found in Kerberos database
Jun 20 11:43:15 dist-test-slave-cjc6 krb5kdc[15206](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.6.234.193@KRBTEST.COM for oryx/127.6.234.254@KRBTEST.COM, Server not found in Kerberos database
I20260620 11:43:15.710414 16260 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 11:43:15.705614 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.234.254:38223 (local address 127.6.234.193:50171)
0620 11:43:15.705785 (+ 171us) negotiation.cc:107] Waiting for socket to connect
0620 11:43:15.705798 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0620 11:43:15.705904 (+ 106us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 11:43:15.706057 (+ 153us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 11:43:15.706057 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 11:43:15.706223 (+ 166us) client_negotiation.cc:189] Negotiated authn=SASL
0620 11:43:15.706269 (+ 46us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 11:43:15.706271 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 11:43:15.706943 (+ 672us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 11:43:15.706945 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 11:43:15.707489 (+ 544us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 11:43:15.707491 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 11:43:15.707866 (+ 375us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 11:43:15.707867 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 11:43:15.707917 (+ 50us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 11:43:15.708382 (+ 465us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 11:43:15.708386 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0620 11:43:15.710316 (+ 1930us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.6.234.254:38223: Server oryx/127.6.234.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":151,"thread_start_us":139,"threads_started":1}
W20260620 11:43:15.710471 16261 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 11:43:15.705716 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.6.234.193:50171 (local address 127.6.234.254:38223)
0620 11:43:15.705833 (+ 117us) server_negotiation.cc:206] Beginning negotiation
0620 11:43:15.705835 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 11:43:15.705885 (+ 50us) server_negotiation.cc:407] Connection header received
0620 11:43:15.705954 (+ 69us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 11:43:15.705954 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 11:43:15.705965 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 11:43:15.706033 (+ 68us) server_negotiation.cc:226] Negotiated authn=SASL
0620 11:43:15.706336 (+ 303us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 11:43:15.706851 (+ 515us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 11:43:15.707596 (+ 745us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 11:43:15.707793 (+ 197us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 11:43:15.707848 (+ 55us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 11:43:15.710400 (+ 2552us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.6.234.193:50171: BlockingRecv error: recv got EOF from 127.6.234.193:50171 (error 108)
Metrics: {"server-negotiator.queue_time_us":97,"thread_start_us":66,"threads_started":1}
Jun 20 11:43:15 dist-test-slave-cjc6 krb5kdc[15206](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.6.234.194@KRBTEST.COM for oryx/127.6.234.254@KRBTEST.COM, Server not found in Kerberos database
Jun 20 11:43:15 dist-test-slave-cjc6 krb5kdc[15206](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.6.234.194@KRBTEST.COM for oryx/127.6.234.254@KRBTEST.COM, Server not found in Kerberos database
I20260620 11:43:15.828510 16262 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 11:43:15.823623 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.234.254:38223 (local address 127.6.234.194:38785)
0620 11:43:15.823762 (+ 139us) negotiation.cc:107] Waiting for socket to connect
0620 11:43:15.823775 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0620 11:43:15.823849 (+ 74us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 11:43:15.823997 (+ 148us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 11:43:15.823998 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 11:43:15.824187 (+ 189us) client_negotiation.cc:189] Negotiated authn=SASL
0620 11:43:15.824232 (+ 45us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 11:43:15.824234 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 11:43:15.824912 (+ 678us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 11:43:15.824913 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 11:43:15.825441 (+ 528us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 11:43:15.825442 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 11:43:15.825881 (+ 439us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 11:43:15.825882 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 11:43:15.825943 (+ 61us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 11:43:15.826398 (+ 455us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 11:43:15.826402 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0620 11:43:15.828417 (+ 2015us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.6.234.254:38223: Server oryx/127.6.234.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":119,"thread_start_us":68,"threads_started":1}
W20260620 11:43:15.828573 16261 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 11:43:15.823664 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.6.234.194:38785 (local address 127.6.234.254:38223)
0620 11:43:15.823722 (+ 58us) server_negotiation.cc:206] Beginning negotiation
0620 11:43:15.823725 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0620 11:43:15.823847 (+ 122us) server_negotiation.cc:407] Connection header received
0620 11:43:15.823923 (+ 76us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 11:43:15.823924 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 11:43:15.823937 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 11:43:15.823983 (+ 46us) server_negotiation.cc:226] Negotiated authn=SASL
0620 11:43:15.824315 (+ 332us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 11:43:15.824818 (+ 503us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 11:43:15.825537 (+ 719us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 11:43:15.825809 (+ 272us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 11:43:15.825883 (+ 74us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 11:43:15.828510 (+ 2627us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.6.234.194:38785: BlockingRecv error: recv got EOF from 127.6.234.194:38785 (error 108)
Metrics: {"server-negotiator.queue_time_us":23}
Jun 20 11:43:15 dist-test-slave-cjc6 krb5kdc[15206](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.6.234.195@KRBTEST.COM for oryx/127.6.234.254@KRBTEST.COM, Server not found in Kerberos database
Jun 20 11:43:15 dist-test-slave-cjc6 krb5kdc[15206](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.6.234.195@KRBTEST.COM for oryx/127.6.234.254@KRBTEST.COM, Server not found in Kerberos database
I20260620 11:43:15.948972 16263 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 11:43:15.944046 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.234.254:38223 (local address 127.6.234.195:46515)
0620 11:43:15.944229 (+ 183us) negotiation.cc:107] Waiting for socket to connect
0620 11:43:15.944241 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0620 11:43:15.944304 (+ 63us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 11:43:15.944431 (+ 127us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 11:43:15.944432 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 11:43:15.944600 (+ 168us) client_negotiation.cc:189] Negotiated authn=SASL
0620 11:43:15.944647 (+ 47us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 11:43:15.944649 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 11:43:15.945287 (+ 638us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 11:43:15.945288 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 11:43:15.945908 (+ 620us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 11:43:15.945912 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 11:43:15.946441 (+ 529us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 11:43:15.946442 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 11:43:15.946483 (+ 41us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 11:43:15.946997 (+ 514us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 11:43:15.947002 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0620 11:43:15.948899 (+ 1897us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.6.234.254:38223: Server oryx/127.6.234.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":153,"thread_start_us":83,"threads_started":1}
W20260620 11:43:15.949087 16261 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 11:43:15.944161 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.6.234.195:46515 (local address 127.6.234.254:38223)
0620 11:43:15.944242 (+ 81us) server_negotiation.cc:206] Beginning negotiation
0620 11:43:15.944244 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 11:43:15.944305 (+ 61us) server_negotiation.cc:407] Connection header received
0620 11:43:15.944349 (+ 44us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 11:43:15.944350 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 11:43:15.944361 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 11:43:15.944411 (+ 50us) server_negotiation.cc:226] Negotiated authn=SASL
0620 11:43:15.944715 (+ 304us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 11:43:15.945192 (+ 477us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 11:43:15.946021 (+ 829us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 11:43:15.946370 (+ 349us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 11:43:15.946432 (+ 62us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 11:43:15.949006 (+ 2574us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.6.234.195:46515: BlockingRecv error: recv got EOF from 127.6.234.195:46515 (error 108)
Metrics: {"server-negotiator.queue_time_us":58}
Jun 20 11:43:16 dist-test-slave-cjc6 krb5kdc[15206](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.6.234.193@KRBTEST.COM for oryx/127.6.234.254@KRBTEST.COM, Server not found in Kerberos database
Jun 20 11:43:16 dist-test-slave-cjc6 krb5kdc[15206](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.6.234.193@KRBTEST.COM for oryx/127.6.234.254@KRBTEST.COM, Server not found in Kerberos database
I20260620 11:43:16.715510 16264 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 11:43:16.711028 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.234.254:38223 (local address 127.6.234.193:32977)
0620 11:43:16.711175 (+ 147us) negotiation.cc:107] Waiting for socket to connect
0620 11:43:16.711189 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0620 11:43:16.711262 (+ 73us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 11:43:16.711431 (+ 169us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 11:43:16.711432 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 11:43:16.711592 (+ 160us) client_negotiation.cc:189] Negotiated authn=SASL
0620 11:43:16.711637 (+ 45us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 11:43:16.711639 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 11:43:16.712286 (+ 647us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 11:43:16.712286 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 11:43:16.712869 (+ 583us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 11:43:16.712870 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 11:43:16.713228 (+ 358us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 11:43:16.713229 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 11:43:16.713266 (+ 37us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 11:43:16.713720 (+ 454us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 11:43:16.713725 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0620 11:43:16.715444 (+ 1719us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.6.234.254:38223: Server oryx/127.6.234.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":126,"thread_start_us":69,"threads_started":1}
W20260620 11:43:16.715608 16265 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 11:43:16.711092 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.6.234.193:32977 (local address 127.6.234.254:38223)
0620 11:43:16.711242 (+ 150us) server_negotiation.cc:206] Beginning negotiation
0620 11:43:16.711244 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 11:43:16.711252 (+ 8us) server_negotiation.cc:407] Connection header received
0620 11:43:16.711309 (+ 57us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 11:43:16.711310 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 11:43:16.711321 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 11:43:16.711378 (+ 57us) server_negotiation.cc:226] Negotiated authn=SASL
0620 11:43:16.711702 (+ 324us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 11:43:16.712218 (+ 516us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 11:43:16.712951 (+ 733us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 11:43:16.713154 (+ 203us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 11:43:16.713212 (+ 58us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 11:43:16.715534 (+ 2322us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.6.234.193:32977: BlockingRecv error: recv got EOF from 127.6.234.193:32977 (error 108)
Metrics: {"server-negotiator.queue_time_us":131,"thread_start_us":80,"threads_started":1}
Jun 20 11:43:16 dist-test-slave-cjc6 krb5kdc[15206](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.6.234.194@KRBTEST.COM for oryx/127.6.234.254@KRBTEST.COM, Server not found in Kerberos database
Jun 20 11:43:16 dist-test-slave-cjc6 krb5kdc[15206](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.6.234.194@KRBTEST.COM for oryx/127.6.234.254@KRBTEST.COM, Server not found in Kerberos database
I20260620 11:43:16.833784 16266 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 11:43:16.829096 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.234.254:38223 (local address 127.6.234.194:47427)
0620 11:43:16.829260 (+ 164us) negotiation.cc:107] Waiting for socket to connect
0620 11:43:16.829270 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0620 11:43:16.829326 (+ 56us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 11:43:16.829477 (+ 151us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 11:43:16.829478 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 11:43:16.829649 (+ 171us) client_negotiation.cc:189] Negotiated authn=SASL
0620 11:43:16.829735 (+ 86us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 11:43:16.829737 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 11:43:16.830431 (+ 694us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 11:43:16.830432 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 11:43:16.831011 (+ 579us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 11:43:16.831012 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 11:43:16.831401 (+ 389us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 11:43:16.831402 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 11:43:16.831468 (+ 66us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 11:43:16.831907 (+ 439us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 11:43:16.831911 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0620 11:43:16.833681 (+ 1770us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.6.234.254:38223: Server oryx/127.6.234.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":147,"thread_start_us":70,"threads_started":1}
W20260620 11:43:16.833877 16265 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 11:43:16.829176 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.6.234.194:47427 (local address 127.6.234.254:38223)
0620 11:43:16.829272 (+ 96us) server_negotiation.cc:206] Beginning negotiation
0620 11:43:16.829273 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0620 11:43:16.829330 (+ 57us) server_negotiation.cc:407] Connection header received
0620 11:43:16.829383 (+ 53us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 11:43:16.829384 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 11:43:16.829395 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 11:43:16.829439 (+ 44us) server_negotiation.cc:226] Negotiated authn=SASL
0620 11:43:16.829823 (+ 384us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 11:43:16.830344 (+ 521us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 11:43:16.831098 (+ 754us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 11:43:16.831313 (+ 215us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 11:43:16.831400 (+ 87us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 11:43:16.833813 (+ 2413us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.6.234.194:47427: BlockingRecv error: recv got EOF from 127.6.234.194:47427 (error 108)
Metrics: {"server-negotiator.queue_time_us":63}
I20260620 11:43:16.871305 7083 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task1JRg7N/build/release/bin/kudu with pid 15789
I20260620 11:43:16.875787 7083 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task1JRg7N/build/release/bin/kudu with pid 15941
I20260620 11:43:16.880404 7083 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task1JRg7N/build/release/bin/kudu with pid 16073
I20260620 11:43:16.884369 7083 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task1JRg7N/build/release/bin/kudu with pid 15719
2026-06-20T11:43:16Z chronyd exiting
[ OK ] SecurityITest.TestMismatchingPrincipals (17438 ms)
[ RUN ] SecurityITest.TestRequireAuthenticationInsecureCluster
2026-06-20T11:43:16Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-20T11:43:16Z Disabled control of system clock
I20260620 11:43:16.911125 7083 external_mini_cluster.cc:1326] Running /tmp/dist-test-task1JRg7N/build/release/bin/kudu
/tmp/dist-test-task1JRg7N/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task1JRg7N/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781955746769051-7083-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task1JRg7N/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781955746769051-7083-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task1JRg7N/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781955746769051-7083-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task1JRg7N/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781955746769051-7083-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.6.234.254:36709
--webserver_interface=127.6.234.254
--webserver_port=0
--builtin_ntp_servers=127.6.234.212:36147
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.6.234.254:36709
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {}
W20260620 11:43:16.989764 16270 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 11:43:16.989964 16270 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 11:43:16.989982 16270 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 11:43:16.991165 16270 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 11:43:16.991209 16270 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 11:43:16.991221 16270 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 11:43:16.991235 16270 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 11:43:16.991284 16270 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 11:43:16.992588 16270 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.6.234.212:36147
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task1JRg7N/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781955746769051-7083-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task1JRg7N/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781955746769051-7083-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.6.234.254:36709
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.6.234.254:36709
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task1JRg7N/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781955746769051-7083-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.6.234.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.16270
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task1JRg7N/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781955746769051-7083-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 20 Jun 2026 10:32:37 UTC on ccycloud.quasar-bgyofv.root.comops.site
build id 4457173
I20260620 11:43:16.992770 16270 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 11:43:16.993072 16270 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 11:43:16.997910 16276 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 11:43:16.997924 16275 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 11:43:16.997987 16270 server_base.cc:1034] running on GCE node
W20260620 11:43:16.997924 16278 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 11:43:16.998292 16270 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 11:43:16.998536 16270 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 11:43:16.999759 16270 hybrid_clock.cc:648] HybridClock initialized: now 1781955796999692 us; error 36 us; skew 500 ppm
I20260620 11:43:17.000898 16270 webserver.cc:460] Webserver started at http://127.6.234.254:34185/ using document root <none> and password file <none>
I20260620 11:43:17.001127 16270 fs_manager.cc:359] Metadata directory not provided
I20260620 11:43:17.001169 16270 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 11:43:17.001318 16270 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 11:43:17.002393 16270 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task1JRg7N/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781955746769051-7083-0/minicluster-data/master-0/data/instance:
uuid: "4ae7376f8aab4728b337fa831d8146f3"
format_stamp: "Formatted at 2026-06-20 11:43:16 on dist-test-slave-cjc6"
server_key: "abc7b8a935505900df37e677b970cde2"
server_key_iv: "1cfd13a121b7c3a1c53e62c243a88284"
server_key_version: "encryptionkey@0"
I20260620 11:43:17.002750 16270 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task1JRg7N/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781955746769051-7083-0/minicluster-data/master-0/wal/instance:
uuid: "4ae7376f8aab4728b337fa831d8146f3"
format_stamp: "Formatted at 2026-06-20 11:43:16 on dist-test-slave-cjc6"
server_key: "abc7b8a935505900df37e677b970cde2"
server_key_iv: "1cfd13a121b7c3a1c53e62c243a88284"
server_key_version: "encryptionkey@0"
I20260620 11:43:17.004057 16270 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260620 11:43:17.004737 16284 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 11:43:17.004926 16270 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260620 11:43:17.005016 16270 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task1JRg7N/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781955746769051-7083-0/minicluster-data/master-0/data,/tmp/dist-test-task1JRg7N/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781955746769051-7083-0/minicluster-data/master-0/wal
uuid: "4ae7376f8aab4728b337fa831d8146f3"
format_stamp: "Formatted at 2026-06-20 11:43:16 on dist-test-slave-cjc6"
server_key: "abc7b8a935505900df37e677b970cde2"
server_key_iv: "1cfd13a121b7c3a1c53e62c243a88284"
server_key_version: "encryptionkey@0"
I20260620 11:43:17.005086 16270 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task1JRg7N/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781955746769051-7083-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task1JRg7N/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781955746769051-7083-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task1JRg7N/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781955746769051-7083-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 11:43:17.021188 16270 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 11:43:17.021488 16270 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 11:43:17.021641 16270 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 11:43:17.025264 16270 rpc_server.cc:307] RPC server started. Bound to: 127.6.234.254:36709
I20260620 11:43:17.025315 16336 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.6.234.254:36709 every 8 connection(s)
I20260620 11:43:17.025620 16270 server_base.cc:1166] Dumped server information to /tmp/dist-test-task1JRg7N/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781955746769051-7083-0/minicluster-data/master-0/data/info.pb
I20260620 11:43:17.026324 16337 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 11:43:17.028810 16337 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 4ae7376f8aab4728b337fa831d8146f3: Bootstrap starting.
I20260620 11:43:17.029395 16337 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 4ae7376f8aab4728b337fa831d8146f3: Neither blocks nor log segments found. Creating new log.
I20260620 11:43:17.029750 16337 log.cc:824] T 00000000000000000000000000000000 P 4ae7376f8aab4728b337fa831d8146f3: Log is configured to *not* fsync() on all Append() calls
I20260620 11:43:17.030421 16337 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 4ae7376f8aab4728b337fa831d8146f3: No bootstrap required, opened a new log
I20260620 11:43:17.031960 16337 raft_consensus.cc:348] T 00000000000000000000000000000000 P 4ae7376f8aab4728b337fa831d8146f3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4ae7376f8aab4728b337fa831d8146f3" member_type: VOTER last_known_addr { host: "127.6.234.254" port: 36709 } }
I20260620 11:43:17.032094 16337 raft_consensus.cc:374] T 00000000000000000000000000000000 P 4ae7376f8aab4728b337fa831d8146f3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 11:43:17.032116 16337 raft_consensus.cc:729] T 00000000000000000000000000000000 P 4ae7376f8aab4728b337fa831d8146f3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4ae7376f8aab4728b337fa831d8146f3, State: Initialized, Role: FOLLOWER
I20260620 11:43:17.032294 16337 consensus_queue.cc:260] T 00000000000000000000000000000000 P 4ae7376f8aab4728b337fa831d8146f3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4ae7376f8aab4728b337fa831d8146f3" member_type: VOTER last_known_addr { host: "127.6.234.254" port: 36709 } }
I20260620 11:43:17.032356 16337 raft_consensus.cc:388] T 00000000000000000000000000000000 P 4ae7376f8aab4728b337fa831d8146f3 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260620 11:43:17.032377 16337 raft_consensus.cc:482] T 00000000000000000000000000000000 P 4ae7376f8aab4728b337fa831d8146f3 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260620 11:43:17.032439 16337 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 4ae7376f8aab4728b337fa831d8146f3 [term 0 FOLLOWER]: Advancing to term 1
I20260620 11:43:17.032958 16337 raft_consensus.cc:504] T 00000000000000000000000000000000 P 4ae7376f8aab4728b337fa831d8146f3 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4ae7376f8aab4728b337fa831d8146f3" member_type: VOTER last_known_addr { host: "127.6.234.254" port: 36709 } }
I20260620 11:43:17.033066 16337 leader_election.cc:302] T 00000000000000000000000000000000 P 4ae7376f8aab4728b337fa831d8146f3 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 4ae7376f8aab4728b337fa831d8146f3; no voters:
I20260620 11:43:17.033244 16337 leader_election.cc:288] T 00000000000000000000000000000000 P 4ae7376f8aab4728b337fa831d8146f3 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260620 11:43:17.033293 16340 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 4ae7376f8aab4728b337fa831d8146f3 [term 1 FOLLOWER]: Leader election won for term 1
I20260620 11:43:17.033484 16340 raft_consensus.cc:686] T 00000000000000000000000000000000 P 4ae7376f8aab4728b337fa831d8146f3 [term 1 LEADER]: Becoming Leader. State: Replica: 4ae7376f8aab4728b337fa831d8146f3, State: Running, Role: LEADER
I20260620 11:43:17.033500 16337 sys_catalog.cc:564] T 00000000000000000000000000000000 P 4ae7376f8aab4728b337fa831d8146f3 [sys.catalog]: configured and running, proceeding with master startup.
I20260620 11:43:17.033629 16340 consensus_queue.cc:237] T 00000000000000000000000000000000 P 4ae7376f8aab4728b337fa831d8146f3 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4ae7376f8aab4728b337fa831d8146f3" member_type: VOTER last_known_addr { host: "127.6.234.254" port: 36709 } }
I20260620 11:43:17.034096 16342 sys_catalog.cc:455] T 00000000000000000000000000000000 P 4ae7376f8aab4728b337fa831d8146f3 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 4ae7376f8aab4728b337fa831d8146f3. Latest consensus state: current_term: 1 leader_uuid: "4ae7376f8aab4728b337fa831d8146f3" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4ae7376f8aab4728b337fa831d8146f3" member_type: VOTER last_known_addr { host: "127.6.234.254" port: 36709 } } }
I20260620 11:43:17.034200 16342 sys_catalog.cc:458] T 00000000000000000000000000000000 P 4ae7376f8aab4728b337fa831d8146f3 [sys.catalog]: This master's current role is: LEADER
I20260620 11:43:17.034227 16341 sys_catalog.cc:455] T 00000000000000000000000000000000 P 4ae7376f8aab4728b337fa831d8146f3 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "4ae7376f8aab4728b337fa831d8146f3" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4ae7376f8aab4728b337fa831d8146f3" member_type: VOTER last_known_addr { host: "127.6.234.254" port: 36709 } } }
I20260620 11:43:17.034312 16341 sys_catalog.cc:458] T 00000000000000000000000000000000 P 4ae7376f8aab4728b337fa831d8146f3 [sys.catalog]: This master's current role is: LEADER
I20260620 11:43:17.034611 16349 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260620 11:43:17.035120 16349 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260620 11:43:17.035212 7083 external_mini_cluster.cc:1388] Started /tmp/dist-test-task1JRg7N/build/release/bin/kudu as pid 16270
I20260620 11:43:17.035320 7083 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task1JRg7N/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781955746769051-7083-0/minicluster-data/master-0/wal/instance
I20260620 11:43:17.035531 7083 external_mini_cluster.cc:1428] Setting key 81ed92831f7a732af51dcc5d935ae7c8
I20260620 11:43:17.037350 16349 catalog_manager.cc:1347] Generated new cluster ID: 7e2bec6840964b0fb5da03f1687cc034
I20260620 11:43:17.037405 16349 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260620 11:43:17.040921 16359 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 11:43:17.036000 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33920 (local address 127.6.234.254:36709)
0620 11:43:17.036574 (+ 574us) server_negotiation.cc:206] Beginning negotiation
0620 11:43:17.036582 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0620 11:43:17.036611 (+ 29us) server_negotiation.cc:407] Connection header received
0620 11:43:17.036743 (+ 132us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 11:43:17.036750 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 11:43:17.036816 (+ 66us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 11:43:17.036951 (+ 135us) server_negotiation.cc:226] Negotiated authn=SASL
0620 11:43:17.037166 (+ 215us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 11:43:17.038233 (+ 1067us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 11:43:17.039027 (+ 794us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 11:43:17.039437 (+ 410us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 11:43:17.039566 (+ 129us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 11:43:17.040266 (+ 700us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 11:43:17.040270 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 11:43:17.040270 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 11:43:17.040445 (+ 175us) server_negotiation.cc:922] calling sasl_server_start()
0620 11:43:17.040477 (+ 32us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 11:43:17.040487 (+ 10us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 11:43:17.040558 (+ 71us) server_negotiation.cc:1035] Waiting for connection context
0620 11:43:17.040635 (+ 77us) server_negotiation.cc:299] Negotiation successful
0620 11:43:17.040712 (+ 77us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":515,"thread_start_us":458,"threads_started":1}
I20260620 11:43:17.072315 16349 catalog_manager.cc:1370] Generated new certificate authority record
I20260620 11:43:17.072861 16349 catalog_manager.cc:1504] Loading token signing keys...
I20260620 11:43:17.077558 16349 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 4ae7376f8aab4728b337fa831d8146f3: Generated new TSK 0
I20260620 11:43:17.077764 16349 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260620 11:43:17.083815 7083 external_mini_cluster.cc:1326] Running /tmp/dist-test-task1JRg7N/build/release/bin/kudu
/tmp/dist-test-task1JRg7N/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task1JRg7N/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781955746769051-7083-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-task1JRg7N/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781955746769051-7083-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task1JRg7N/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781955746769051-7083-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-task1JRg7N/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781955746769051-7083-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.6.234.193:0
--local_ip_for_outbound_sockets=127.6.234.193
--webserver_interface=127.6.234.193
--webserver_port=0
--tserver_master_addrs=127.6.234.254:36709
--builtin_ntp_servers=127.6.234.212:36147
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {}
W20260620 11:43:17.185489 16361 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 11:43:17.185686 16361 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 11:43:17.185722 16361 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 11:43:17.186872 16361 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 11:43:17.186910 16361 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 11:43:17.186935 16361 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.6.234.193
I20260620 11:43:17.188241 16361 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.6.234.212:36147
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task1JRg7N/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781955746769051-7083-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-task1JRg7N/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781955746769051-7083-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.6.234.193:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task1JRg7N/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781955746769051-7083-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.6.234.193
--webserver_port=0
--tserver_master_addrs=127.6.234.254:36709
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.16361
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.6.234.193
--log_dir=/tmp/dist-test-task1JRg7N/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781955746769051-7083-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 20 Jun 2026 10:32:37 UTC on ccycloud.quasar-bgyofv.root.comops.site
build id 4457173
I20260620 11:43:17.188486 16361 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 11:43:17.188769 16361 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 11:43:17.193568 16369 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 11:43:17.193547 16366 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 11:43:17.193780 16361 server_base.cc:1034] running on GCE node
W20260620 11:43:17.193535 16367 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 11:43:17.194135 16361 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 11:43:17.194419 16361 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 11:43:17.195577 16361 hybrid_clock.cc:648] HybridClock initialized: now 1781955797195562 us; error 32 us; skew 500 ppm
I20260620 11:43:17.196784 16361 webserver.cc:460] Webserver started at http://127.6.234.193:33823/ using document root <none> and password file <none>
I20260620 11:43:17.197031 16361 fs_manager.cc:359] Metadata directory not provided
I20260620 11:43:17.197078 16361 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 11:43:17.197201 16361 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 11:43:17.198297 16361 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task1JRg7N/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781955746769051-7083-0/minicluster-data/ts-0/data/instance:
uuid: "d99ad3ca2556404794f96e8e573bf5a4"
format_stamp: "Formatted at 2026-06-20 11:43:17 on dist-test-slave-cjc6"
server_key: "57bb06ea0b90b07fc44a4a2e2a7fb85b"
server_key_iv: "366215fc93a303d51714c31eba8be778"
server_key_version: "encryptionkey@0"
I20260620 11:43:17.198669 16361 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task1JRg7N/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781955746769051-7083-0/minicluster-data/ts-0/wal/instance:
uuid: "d99ad3ca2556404794f96e8e573bf5a4"
format_stamp: "Formatted at 2026-06-20 11:43:17 on dist-test-slave-cjc6"
server_key: "57bb06ea0b90b07fc44a4a2e2a7fb85b"
server_key_iv: "366215fc93a303d51714c31eba8be778"
server_key_version: "encryptionkey@0"
I20260620 11:43:17.199913 16361 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260620 11:43:17.200618 16375 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 11:43:17.200790 16361 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260620 11:43:17.200887 16361 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task1JRg7N/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781955746769051-7083-0/minicluster-data/ts-0/data,/tmp/dist-test-task1JRg7N/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781955746769051-7083-0/minicluster-data/ts-0/wal
uuid: "d99ad3ca2556404794f96e8e573bf5a4"
format_stamp: "Formatted at 2026-06-20 11:43:17 on dist-test-slave-cjc6"
server_key: "57bb06ea0b90b07fc44a4a2e2a7fb85b"
server_key_iv: "366215fc93a303d51714c31eba8be778"
server_key_version: "encryptionkey@0"
I20260620 11:43:17.200968 16361 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task1JRg7N/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781955746769051-7083-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-task1JRg7N/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781955746769051-7083-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-task1JRg7N/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781955746769051-7083-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 11:43:17.218050 16361 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 11:43:17.218333 16361 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 11:43:17.218482 16361 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 11:43:17.218708 16361 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 11:43:17.219024 16361 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 11:43:17.219079 16361 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 11:43:17.219113 16361 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 11:43:17.219142 16361 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 11:43:17.224846 16361 rpc_server.cc:307] RPC server started. Bound to: 127.6.234.193:46339
I20260620 11:43:17.224882 16488 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.6.234.193:46339 every 8 connection(s)
I20260620 11:43:17.225191 16361 server_base.cc:1166] Dumped server information to /tmp/dist-test-task1JRg7N/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781955746769051-7083-0/minicluster-data/ts-0/data/info.pb
I20260620 11:43:17.229501 7083 external_mini_cluster.cc:1388] Started /tmp/dist-test-task1JRg7N/build/release/bin/kudu as pid 16361
I20260620 11:43:17.229601 7083 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task1JRg7N/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781955746769051-7083-0/minicluster-data/ts-0/wal/instance
I20260620 11:43:17.229872 7083 external_mini_cluster.cc:1428] Setting key 7d912cc021ba9a55ee60600400559271
I20260620 11:43:17.230216 16359 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 11:43:17.225896 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.6.234.193:37929 (local address 127.6.234.254:36709)
0620 11:43:17.225956 (+ 60us) server_negotiation.cc:206] Beginning negotiation
0620 11:43:17.225958 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 11:43:17.226178 (+ 220us) server_negotiation.cc:407] Connection header received
0620 11:43:17.226631 (+ 453us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 11:43:17.226632 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 11:43:17.226642 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 11:43:17.226699 (+ 57us) server_negotiation.cc:226] Negotiated authn=SASL
0620 11:43:17.227281 (+ 582us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 11:43:17.227761 (+ 480us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 11:43:17.228661 (+ 900us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 11:43:17.228840 (+ 179us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 11:43:17.228891 (+ 51us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 11:43:17.229897 (+ 1006us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 11:43:17.229899 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 11:43:17.229899 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 11:43:17.229907 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0620 11:43:17.229926 (+ 19us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 11:43:17.229931 (+ 5us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 11:43:17.229994 (+ 63us) server_negotiation.cc:1035] Waiting for connection context
0620 11:43:17.230097 (+ 103us) server_negotiation.cc:299] Negotiation successful
0620 11:43:17.230127 (+ 30us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":36}
I20260620 11:43:17.230453 16491 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 11:43:17.225869 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.234.254:36709 (local address 127.6.234.193:37929)
0620 11:43:17.226046 (+ 177us) negotiation.cc:107] Waiting for socket to connect
0620 11:43:17.226086 (+ 40us) client_negotiation.cc:174] Beginning negotiation
0620 11:43:17.226552 (+ 466us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 11:43:17.226748 (+ 196us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 11:43:17.226752 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 11:43:17.226760 (+ 8us) client_negotiation.cc:189] Negotiated authn=SASL
0620 11:43:17.227216 (+ 456us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 11:43:17.227220 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 11:43:17.227837 (+ 617us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 11:43:17.227838 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 11:43:17.228587 (+ 749us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 11:43:17.228589 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 11:43:17.228938 (+ 349us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 11:43:17.228939 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 11:43:17.229053 (+ 114us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 11:43:17.229729 (+ 676us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0620 11:43:17.229730 (+ 1us) client_negotiation.cc:656] calling sasl_client_start()
0620 11:43:17.229779 (+ 49us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0620 11:43:17.229802 (+ 23us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 11:43:17.230021 (+ 219us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 11:43:17.230027 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 11:43:17.230035 (+ 8us) client_negotiation.cc:769] Sending connection context
0620 11:43:17.230092 (+ 57us) client_negotiation.cc:240] Negotiation successful
0620 11:43:17.230214 (+ 122us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":112,"thread_start_us":83,"threads_started":1}
I20260620 11:43:17.230818 7083 external_mini_cluster.cc:1326] Running /tmp/dist-test-task1JRg7N/build/release/bin/kudu
/tmp/dist-test-task1JRg7N/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task1JRg7N/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781955746769051-7083-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-task1JRg7N/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781955746769051-7083-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task1JRg7N/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781955746769051-7083-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-task1JRg7N/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781955746769051-7083-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.6.234.194:0
--local_ip_for_outbound_sockets=127.6.234.194
--webserver_interface=127.6.234.194
--webserver_port=0
--tserver_master_addrs=127.6.234.254:36709
--builtin_ntp_servers=127.6.234.212:36147
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {}
I20260620 11:43:17.230916 16489 heartbeater.cc:347] Connected to a master server at 127.6.234.254:36709
I20260620 11:43:17.230995 16489 heartbeater.cc:464] Registering TS with master...
I20260620 11:43:17.231184 16489 heartbeater.cc:511] Master 127.6.234.254:36709 requested a full tablet report, sending...
I20260620 11:43:17.231671 16301 ts_manager.cc:194] Registered new tserver with Master: d99ad3ca2556404794f96e8e573bf5a4 (127.6.234.193:46339)
I20260620 11:43:17.232590 16301 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.6.234.193:37929
W20260620 11:43:17.310254 16492 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 11:43:17.310436 16492 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 11:43:17.310456 16492 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 11:43:17.311486 16492 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 11:43:17.311517 16492 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 11:43:17.311542 16492 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.6.234.194
I20260620 11:43:17.312760 16492 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.6.234.212:36147
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task1JRg7N/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781955746769051-7083-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-task1JRg7N/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781955746769051-7083-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.6.234.194:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task1JRg7N/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781955746769051-7083-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.6.234.194
--webserver_port=0
--tserver_master_addrs=127.6.234.254:36709
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.16492
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.6.234.194
--log_dir=/tmp/dist-test-task1JRg7N/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781955746769051-7083-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 20 Jun 2026 10:32:37 UTC on ccycloud.quasar-bgyofv.root.comops.site
build id 4457173
I20260620 11:43:17.312925 16492 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 11:43:17.313144 16492 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 11:43:17.317770 16500 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 11:43:17.317857 16492 server_base.cc:1034] running on GCE node
W20260620 11:43:17.317795 16498 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 11:43:17.317806 16497 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 11:43:17.318234 16492 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 11:43:17.318472 16492 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 11:43:17.319618 16492 hybrid_clock.cc:648] HybridClock initialized: now 1781955797319599 us; error 36 us; skew 500 ppm
I20260620 11:43:17.320851 16492 webserver.cc:460] Webserver started at http://127.6.234.194:38403/ using document root <none> and password file <none>
I20260620 11:43:17.321089 16492 fs_manager.cc:359] Metadata directory not provided
I20260620 11:43:17.321134 16492 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 11:43:17.321303 16492 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 11:43:17.322355 16492 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task1JRg7N/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781955746769051-7083-0/minicluster-data/ts-1/data/instance:
uuid: "a528ba9fd53447ce86928432b3693103"
format_stamp: "Formatted at 2026-06-20 11:43:17 on dist-test-slave-cjc6"
server_key: "99f0f1e022580ccffcb3175cdb635358"
server_key_iv: "aa7dfecf4000e233155e1ce17b633d44"
server_key_version: "encryptionkey@0"
I20260620 11:43:17.322692 16492 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task1JRg7N/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781955746769051-7083-0/minicluster-data/ts-1/wal/instance:
uuid: "a528ba9fd53447ce86928432b3693103"
format_stamp: "Formatted at 2026-06-20 11:43:17 on dist-test-slave-cjc6"
server_key: "99f0f1e022580ccffcb3175cdb635358"
server_key_iv: "aa7dfecf4000e233155e1ce17b633d44"
server_key_version: "encryptionkey@0"
I20260620 11:43:17.323866 16492 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260620 11:43:17.324522 16506 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 11:43:17.324678 16492 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260620 11:43:17.324790 16492 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task1JRg7N/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781955746769051-7083-0/minicluster-data/ts-1/data,/tmp/dist-test-task1JRg7N/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781955746769051-7083-0/minicluster-data/ts-1/wal
uuid: "a528ba9fd53447ce86928432b3693103"
format_stamp: "Formatted at 2026-06-20 11:43:17 on dist-test-slave-cjc6"
server_key: "99f0f1e022580ccffcb3175cdb635358"
server_key_iv: "aa7dfecf4000e233155e1ce17b633d44"
server_key_version: "encryptionkey@0"
I20260620 11:43:17.324868 16492 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task1JRg7N/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781955746769051-7083-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-task1JRg7N/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781955746769051-7083-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-task1JRg7N/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781955746769051-7083-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 11:43:17.339174 16492 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 11:43:17.339433 16492 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 11:43:17.339591 16492 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 11:43:17.339797 16492 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 11:43:17.340098 16492 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 11:43:17.340154 16492 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 11:43:17.340198 16492 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 11:43:17.340226 16492 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 11:43:17.345599 16492 rpc_server.cc:307] RPC server started. Bound to: 127.6.234.194:33111
I20260620 11:43:17.345674 16619 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.6.234.194:33111 every 8 connection(s)
I20260620 11:43:17.346021 16492 server_base.cc:1166] Dumped server information to /tmp/dist-test-task1JRg7N/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781955746769051-7083-0/minicluster-data/ts-1/data/info.pb
I20260620 11:43:17.350800 16359 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 11:43:17.346710 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.6.234.194:33181 (local address 127.6.234.254:36709)
0620 11:43:17.346789 (+ 79us) server_negotiation.cc:206] Beginning negotiation
0620 11:43:17.346792 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0620 11:43:17.347090 (+ 298us) server_negotiation.cc:407] Connection header received
0620 11:43:17.347543 (+ 453us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 11:43:17.347545 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 11:43:17.347555 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 11:43:17.347605 (+ 50us) server_negotiation.cc:226] Negotiated authn=SASL
0620 11:43:17.348220 (+ 615us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 11:43:17.348696 (+ 476us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 11:43:17.349597 (+ 901us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 11:43:17.349845 (+ 248us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 11:43:17.349896 (+ 51us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 11:43:17.350590 (+ 694us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 11:43:17.350591 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 11:43:17.350591 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 11:43:17.350595 (+ 4us) server_negotiation.cc:922] calling sasl_server_start()
0620 11:43:17.350608 (+ 13us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 11:43:17.350612 (+ 4us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 11:43:17.350663 (+ 51us) server_negotiation.cc:1035] Waiting for connection context
0620 11:43:17.350724 (+ 61us) server_negotiation.cc:299] Negotiation successful
0620 11:43:17.350745 (+ 21us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":45}
I20260620 11:43:17.350962 16622 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 11:43:17.346689 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.234.254:36709 (local address 127.6.234.194:33181)
0620 11:43:17.346945 (+ 256us) negotiation.cc:107] Waiting for socket to connect
0620 11:43:17.346989 (+ 44us) client_negotiation.cc:174] Beginning negotiation
0620 11:43:17.347455 (+ 466us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 11:43:17.347657 (+ 202us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 11:43:17.347660 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 11:43:17.347670 (+ 10us) client_negotiation.cc:189] Negotiated authn=SASL
0620 11:43:17.348129 (+ 459us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 11:43:17.348133 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 11:43:17.348770 (+ 637us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 11:43:17.348771 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 11:43:17.349523 (+ 752us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 11:43:17.349524 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 11:43:17.349915 (+ 391us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 11:43:17.349916 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 11:43:17.350018 (+ 102us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 11:43:17.350465 (+ 447us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0620 11:43:17.350465 (+ 0us) client_negotiation.cc:656] calling sasl_client_start()
0620 11:43:17.350492 (+ 27us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0620 11:43:17.350513 (+ 21us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 11:43:17.350664 (+ 151us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 11:43:17.350666 (+ 2us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 11:43:17.350668 (+ 2us) client_negotiation.cc:769] Sending connection context
0620 11:43:17.350712 (+ 44us) client_negotiation.cc:240] Negotiation successful
0620 11:43:17.350780 (+ 68us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":174,"thread_start_us":110,"threads_started":1}
I20260620 11:43:17.351438 16620 heartbeater.cc:347] Connected to a master server at 127.6.234.254:36709
I20260620 11:43:17.351512 16620 heartbeater.cc:464] Registering TS with master...
I20260620 11:43:17.351643 16620 heartbeater.cc:511] Master 127.6.234.254:36709 requested a full tablet report, sending...
I20260620 11:43:17.351959 16301 ts_manager.cc:194] Registered new tserver with Master: a528ba9fd53447ce86928432b3693103 (127.6.234.194:33111)
I20260620 11:43:17.352290 16301 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.6.234.194:33181
I20260620 11:43:17.355732 7083 external_mini_cluster.cc:1388] Started /tmp/dist-test-task1JRg7N/build/release/bin/kudu as pid 16492
I20260620 11:43:17.355806 7083 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task1JRg7N/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781955746769051-7083-0/minicluster-data/ts-1/wal/instance
I20260620 11:43:17.355947 7083 external_mini_cluster.cc:1428] Setting key b3dadbca087226e5d6993d76f1497972
I20260620 11:43:17.356726 7083 external_mini_cluster.cc:1326] Running /tmp/dist-test-task1JRg7N/build/release/bin/kudu
/tmp/dist-test-task1JRg7N/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task1JRg7N/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781955746769051-7083-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-task1JRg7N/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781955746769051-7083-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task1JRg7N/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781955746769051-7083-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-task1JRg7N/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781955746769051-7083-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.6.234.195:0
--local_ip_for_outbound_sockets=127.6.234.195
--webserver_interface=127.6.234.195
--webserver_port=0
--tserver_master_addrs=127.6.234.254:36709
--builtin_ntp_servers=127.6.234.212:36147
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {}
W20260620 11:43:17.438647 16623 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 11:43:17.438869 16623 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 11:43:17.438903 16623 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 11:43:17.440052 16623 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 11:43:17.440114 16623 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 11:43:17.440157 16623 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.6.234.195
I20260620 11:43:17.441560 16623 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.6.234.212:36147
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task1JRg7N/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781955746769051-7083-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-task1JRg7N/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781955746769051-7083-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.6.234.195:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task1JRg7N/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781955746769051-7083-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.6.234.195
--webserver_port=0
--tserver_master_addrs=127.6.234.254:36709
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.16623
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.6.234.195
--log_dir=/tmp/dist-test-task1JRg7N/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781955746769051-7083-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 20 Jun 2026 10:32:37 UTC on ccycloud.quasar-bgyofv.root.comops.site
build id 4457173
I20260620 11:43:17.441865 16623 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 11:43:17.442164 16623 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 11:43:17.448700 16628 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 11:43:17.448706 16629 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 11:43:17.448822 16631 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 11:43:17.449258 16623 server_base.cc:1034] running on GCE node
I20260620 11:43:17.449450 16623 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 11:43:17.449682 16623 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 11:43:17.450862 16623 hybrid_clock.cc:648] HybridClock initialized: now 1781955797450841 us; error 41 us; skew 500 ppm
I20260620 11:43:17.452075 16623 webserver.cc:460] Webserver started at http://127.6.234.195:37437/ using document root <none> and password file <none>
I20260620 11:43:17.452332 16623 fs_manager.cc:359] Metadata directory not provided
I20260620 11:43:17.452404 16623 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 11:43:17.452559 16623 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 11:43:17.453598 16623 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task1JRg7N/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781955746769051-7083-0/minicluster-data/ts-2/data/instance:
uuid: "5040b0d216414e08ab0cb7551c7d9b17"
format_stamp: "Formatted at 2026-06-20 11:43:17 on dist-test-slave-cjc6"
server_key: "eadb4e3981133d0ed58192c7002b74d6"
server_key_iv: "989e724300755ab475cbb8a370a6d6d4"
server_key_version: "encryptionkey@0"
I20260620 11:43:17.454012 16623 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task1JRg7N/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781955746769051-7083-0/minicluster-data/ts-2/wal/instance:
uuid: "5040b0d216414e08ab0cb7551c7d9b17"
format_stamp: "Formatted at 2026-06-20 11:43:17 on dist-test-slave-cjc6"
server_key: "eadb4e3981133d0ed58192c7002b74d6"
server_key_iv: "989e724300755ab475cbb8a370a6d6d4"
server_key_version: "encryptionkey@0"
I20260620 11:43:17.455348 16623 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.001s sys 0.000s
I20260620 11:43:17.456126 16637 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 11:43:17.456306 16623 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260620 11:43:17.456400 16623 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task1JRg7N/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781955746769051-7083-0/minicluster-data/ts-2/data,/tmp/dist-test-task1JRg7N/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781955746769051-7083-0/minicluster-data/ts-2/wal
uuid: "5040b0d216414e08ab0cb7551c7d9b17"
format_stamp: "Formatted at 2026-06-20 11:43:17 on dist-test-slave-cjc6"
server_key: "eadb4e3981133d0ed58192c7002b74d6"
server_key_iv: "989e724300755ab475cbb8a370a6d6d4"
server_key_version: "encryptionkey@0"
I20260620 11:43:17.456481 16623 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task1JRg7N/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781955746769051-7083-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-task1JRg7N/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781955746769051-7083-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-task1JRg7N/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781955746769051-7083-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 11:43:17.475365 16623 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 11:43:17.475698 16623 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 11:43:17.475865 16623 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 11:43:17.476084 16623 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 11:43:17.476405 16623 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 11:43:17.476460 16623 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 11:43:17.476503 16623 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 11:43:17.476531 16623 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 11:43:17.482772 16623 rpc_server.cc:307] RPC server started. Bound to: 127.6.234.195:45581
I20260620 11:43:17.482813 16750 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.6.234.195:45581 every 8 connection(s)
I20260620 11:43:17.483134 16623 server_base.cc:1166] Dumped server information to /tmp/dist-test-task1JRg7N/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781955746769051-7083-0/minicluster-data/ts-2/data/info.pb
I20260620 11:43:17.489521 16359 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 11:43:17.483840 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.6.234.195:46127 (local address 127.6.234.254:36709)
0620 11:43:17.483913 (+ 73us) server_negotiation.cc:206] Beginning negotiation
0620 11:43:17.483915 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 11:43:17.484217 (+ 302us) server_negotiation.cc:407] Connection header received
0620 11:43:17.485031 (+ 814us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 11:43:17.485031 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 11:43:17.485041 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 11:43:17.485085 (+ 44us) server_negotiation.cc:226] Negotiated authn=SASL
0620 11:43:17.486010 (+ 925us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 11:43:17.486496 (+ 486us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 11:43:17.488134 (+ 1638us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 11:43:17.488315 (+ 181us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 11:43:17.488362 (+ 47us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 11:43:17.489291 (+ 929us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 11:43:17.489292 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 11:43:17.489292 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 11:43:17.489296 (+ 4us) server_negotiation.cc:922] calling sasl_server_start()
0620 11:43:17.489309 (+ 13us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 11:43:17.489313 (+ 4us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 11:43:17.489353 (+ 40us) server_negotiation.cc:1035] Waiting for connection context
0620 11:43:17.489438 (+ 85us) server_negotiation.cc:299] Negotiation successful
0620 11:43:17.489463 (+ 25us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":42}
I20260620 11:43:17.489925 16753 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 11:43:17.483827 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.234.254:36709 (local address 127.6.234.195:46127)
0620 11:43:17.484068 (+ 241us) negotiation.cc:107] Waiting for socket to connect
0620 11:43:17.484116 (+ 48us) client_negotiation.cc:174] Beginning negotiation
0620 11:43:17.484930 (+ 814us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 11:43:17.485164 (+ 234us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 11:43:17.485168 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 11:43:17.485180 (+ 12us) client_negotiation.cc:189] Negotiated authn=SASL
0620 11:43:17.485911 (+ 731us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 11:43:17.485918 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 11:43:17.486580 (+ 662us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 11:43:17.486581 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 11:43:17.488042 (+ 1461us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 11:43:17.488045 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 11:43:17.488391 (+ 346us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 11:43:17.488392 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 11:43:17.488565 (+ 173us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 11:43:17.489127 (+ 562us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0620 11:43:17.489128 (+ 1us) client_negotiation.cc:656] calling sasl_client_start()
0620 11:43:17.489170 (+ 42us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0620 11:43:17.489200 (+ 30us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 11:43:17.489356 (+ 156us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 11:43:17.489361 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 11:43:17.489364 (+ 3us) client_negotiation.cc:769] Sending connection context
0620 11:43:17.489433 (+ 69us) client_negotiation.cc:240] Negotiation successful
0620 11:43:17.489550 (+ 117us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":154,"thread_start_us":103,"threads_started":1}
I20260620 11:43:17.490442 16751 heartbeater.cc:347] Connected to a master server at 127.6.234.254:36709
I20260620 11:43:17.490566 16751 heartbeater.cc:464] Registering TS with master...
I20260620 11:43:17.490746 16751 heartbeater.cc:511] Master 127.6.234.254:36709 requested a full tablet report, sending...
I20260620 11:43:17.491181 16301 ts_manager.cc:194] Registered new tserver with Master: 5040b0d216414e08ab0cb7551c7d9b17 (127.6.234.195:45581)
I20260620 11:43:17.491276 7083 external_mini_cluster.cc:1388] Started /tmp/dist-test-task1JRg7N/build/release/bin/kudu as pid 16623
I20260620 11:43:17.491364 7083 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task1JRg7N/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781955746769051-7083-0/minicluster-data/ts-2/wal/instance
I20260620 11:43:17.491554 7083 external_mini_cluster.cc:1428] Setting key c0f16413ab391724ffabb8ed2a015efc
I20260620 11:43:17.491681 16301 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.6.234.195:46127
I20260620 11:43:17.492394 7083 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
W20260620 11:43:17.494352 16359 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 11:43:17.493803 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33934 (local address 127.6.234.254:36709)
0620 11:43:17.493840 (+ 37us) server_negotiation.cc:206] Beginning negotiation
0620 11:43:17.493841 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0620 11:43:17.494047 (+ 206us) server_negotiation.cc:407] Connection header received
0620 11:43:17.494073 (+ 26us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 11:43:17.494074 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 11:43:17.494090 (+ 16us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 11:43:17.494149 (+ 59us) server_negotiation.cc:226] Negotiated authn=SASL
0620 11:43:17.494277 (+ 128us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:33934: BlockingRecv error: recv got EOF from 127.0.0.1:33934 (error 108)
Metrics: {"server-negotiator.queue_time_us":15}
I20260620 11:43:17.494374 7083 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task1JRg7N/build/release/bin/kudu with pid 16361
I20260620 11:43:17.499008 7083 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task1JRg7N/build/release/bin/kudu with pid 16492
I20260620 11:43:17.503983 7083 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task1JRg7N/build/release/bin/kudu with pid 16623
I20260620 11:43:17.508715 7083 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task1JRg7N/build/release/bin/kudu with pid 16270
2026-06-20T11:43:17Z chronyd exiting
[ OK ] SecurityITest.TestRequireAuthenticationInsecureCluster (623 ms)
[ RUN ] SecurityITest.TestRequireEncryptionInsecureCluster
2026-06-20T11:43:17Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-20T11:43:17Z Disabled control of system clock
I20260620 11:43:17.535074 7083 external_mini_cluster.cc:1326] Running /tmp/dist-test-task1JRg7N/build/release/bin/kudu
/tmp/dist-test-task1JRg7N/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task1JRg7N/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781955746769051-7083-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task1JRg7N/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781955746769051-7083-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task1JRg7N/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781955746769051-7083-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task1JRg7N/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781955746769051-7083-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.6.234.254:45627
--webserver_interface=127.6.234.254
--webserver_port=0
--builtin_ntp_servers=127.6.234.212:34467
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.6.234.254:45627
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--rpc_encryption=disabled
--rpc_authentication=disabled with env {}
W20260620 11:43:17.614825 16762 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 11:43:17.615028 16762 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 11:43:17.615048 16762 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 11:43:17.616113 16762 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 11:43:17.616146 16762 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 11:43:17.616159 16762 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 11:43:17.616173 16762 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 11:43:17.616184 16762 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 11:43:17.617426 16762 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.6.234.212:34467
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task1JRg7N/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781955746769051-7083-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task1JRg7N/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781955746769051-7083-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.6.234.254:45627
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.6.234.254:45627
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task1JRg7N/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781955746769051-7083-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.6.234.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.16762
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task1JRg7N/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781955746769051-7083-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 20 Jun 2026 10:32:37 UTC on ccycloud.quasar-bgyofv.root.comops.site
build id 4457173
I20260620 11:43:17.617624 16762 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 11:43:17.617944 16762 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 11:43:17.622579 16767 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 11:43:17.622648 16770 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 11:43:17.622598 16768 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 11:43:17.622826 16762 server_base.cc:1034] running on GCE node
I20260620 11:43:17.623054 16762 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 11:43:17.623343 16762 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 11:43:17.624526 16762 hybrid_clock.cc:648] HybridClock initialized: now 1781955797624422 us; error 113 us; skew 500 ppm
I20260620 11:43:17.625792 16762 webserver.cc:460] Webserver started at http://127.6.234.254:43327/ using document root <none> and password file <none>
I20260620 11:43:17.626047 16762 fs_manager.cc:359] Metadata directory not provided
I20260620 11:43:17.626094 16762 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 11:43:17.626257 16762 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 11:43:17.627367 16762 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task1JRg7N/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781955746769051-7083-0/minicluster-data/master-0/data/instance:
uuid: "e0f9300b3f064e4faaa072a48385ff8e"
format_stamp: "Formatted at 2026-06-20 11:43:17 on dist-test-slave-cjc6"
server_key: "82474ff009deea46d6067cec4b18a11e"
server_key_iv: "632b285252e358d4a521105dbce072e4"
server_key_version: "encryptionkey@0"
I20260620 11:43:17.627743 16762 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task1JRg7N/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781955746769051-7083-0/minicluster-data/master-0/wal/instance:
uuid: "e0f9300b3f064e4faaa072a48385ff8e"
format_stamp: "Formatted at 2026-06-20 11:43:17 on dist-test-slave-cjc6"
server_key: "82474ff009deea46d6067cec4b18a11e"
server_key_iv: "632b285252e358d4a521105dbce072e4"
server_key_version: "encryptionkey@0"
I20260620 11:43:17.629074 16762 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.003s
I20260620 11:43:17.629833 16776 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 11:43:17.630007 16762 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.001s
I20260620 11:43:17.630070 16762 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task1JRg7N/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781955746769051-7083-0/minicluster-data/master-0/data,/tmp/dist-test-task1JRg7N/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781955746769051-7083-0/minicluster-data/master-0/wal
uuid: "e0f9300b3f064e4faaa072a48385ff8e"
format_stamp: "Formatted at 2026-06-20 11:43:17 on dist-test-slave-cjc6"
server_key: "82474ff009deea46d6067cec4b18a11e"
server_key_iv: "632b285252e358d4a521105dbce072e4"
server_key_version: "encryptionkey@0"
I20260620 11:43:17.630162 16762 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task1JRg7N/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781955746769051-7083-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task1JRg7N/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781955746769051-7083-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task1JRg7N/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781955746769051-7083-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 11:43:17.633370 16762 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 11:43:17.633628 16762 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 11:43:17.633785 16762 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 11:43:17.637423 16762 rpc_server.cc:307] RPC server started. Bound to: 127.6.234.254:45627
I20260620 11:43:17.637454 16828 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.6.234.254:45627 every 8 connection(s)
I20260620 11:43:17.637825 16762 server_base.cc:1166] Dumped server information to /tmp/dist-test-task1JRg7N/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781955746769051-7083-0/minicluster-data/master-0/data/info.pb
I20260620 11:43:17.638474 16829 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 11:43:17.638919 7083 external_mini_cluster.cc:1388] Started /tmp/dist-test-task1JRg7N/build/release/bin/kudu as pid 16762
I20260620 11:43:17.639029 7083 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task1JRg7N/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781955746769051-7083-0/minicluster-data/master-0/wal/instance
I20260620 11:43:17.639233 7083 external_mini_cluster.cc:1428] Setting key a86d65da23f4c06cfc2c56c661328b34
I20260620 11:43:17.641779 16832 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 11:43:17.639783 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58580 (local address 127.6.234.254:45627)
0620 11:43:17.639990 (+ 207us) server_negotiation.cc:206] Beginning negotiation
0620 11:43:17.639996 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0620 11:43:17.640024 (+ 28us) server_negotiation.cc:407] Connection header received
0620 11:43:17.640473 (+ 449us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 11:43:17.640479 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 11:43:17.640515 (+ 36us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 11:43:17.640680 (+ 165us) server_negotiation.cc:226] Negotiated authn=SASL
0620 11:43:17.641324 (+ 644us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 11:43:17.641327 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 11:43:17.641328 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 11:43:17.641332 (+ 4us) server_negotiation.cc:922] calling sasl_server_start()
0620 11:43:17.641380 (+ 48us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 11:43:17.641390 (+ 10us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 11:43:17.641464 (+ 74us) server_negotiation.cc:1035] Waiting for connection context
0620 11:43:17.641529 (+ 65us) server_negotiation.cc:299] Negotiation successful
0620 11:43:17.641591 (+ 62us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":128,"thread_start_us":80,"threads_started":1}
I20260620 11:43:17.642181 16829 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P e0f9300b3f064e4faaa072a48385ff8e: Bootstrap starting.
I20260620 11:43:17.642827 16829 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P e0f9300b3f064e4faaa072a48385ff8e: Neither blocks nor log segments found. Creating new log.
I20260620 11:43:17.643044 16829 log.cc:824] T 00000000000000000000000000000000 P e0f9300b3f064e4faaa072a48385ff8e: Log is configured to *not* fsync() on all Append() calls
I20260620 11:43:17.643637 16829 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P e0f9300b3f064e4faaa072a48385ff8e: No bootstrap required, opened a new log
I20260620 11:43:17.644869 16829 raft_consensus.cc:348] T 00000000000000000000000000000000 P e0f9300b3f064e4faaa072a48385ff8e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e0f9300b3f064e4faaa072a48385ff8e" member_type: VOTER last_known_addr { host: "127.6.234.254" port: 45627 } }
I20260620 11:43:17.644999 16829 raft_consensus.cc:374] T 00000000000000000000000000000000 P e0f9300b3f064e4faaa072a48385ff8e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 11:43:17.645022 16829 raft_consensus.cc:729] T 00000000000000000000000000000000 P e0f9300b3f064e4faaa072a48385ff8e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e0f9300b3f064e4faaa072a48385ff8e, State: Initialized, Role: FOLLOWER
I20260620 11:43:17.645139 16829 consensus_queue.cc:260] T 00000000000000000000000000000000 P e0f9300b3f064e4faaa072a48385ff8e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e0f9300b3f064e4faaa072a48385ff8e" member_type: VOTER last_known_addr { host: "127.6.234.254" port: 45627 } }
I20260620 11:43:17.645233 16829 raft_consensus.cc:388] T 00000000000000000000000000000000 P e0f9300b3f064e4faaa072a48385ff8e [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260620 11:43:17.645255 16829 raft_consensus.cc:482] T 00000000000000000000000000000000 P e0f9300b3f064e4faaa072a48385ff8e [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260620 11:43:17.645282 16829 raft_consensus.cc:3037] T 00000000000000000000000000000000 P e0f9300b3f064e4faaa072a48385ff8e [term 0 FOLLOWER]: Advancing to term 1
I20260620 11:43:17.645808 16829 raft_consensus.cc:504] T 00000000000000000000000000000000 P e0f9300b3f064e4faaa072a48385ff8e [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e0f9300b3f064e4faaa072a48385ff8e" member_type: VOTER last_known_addr { host: "127.6.234.254" port: 45627 } }
I20260620 11:43:17.645892 16829 leader_election.cc:302] T 00000000000000000000000000000000 P e0f9300b3f064e4faaa072a48385ff8e [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: e0f9300b3f064e4faaa072a48385ff8e; no voters:
I20260620 11:43:17.646008 16829 leader_election.cc:288] T 00000000000000000000000000000000 P e0f9300b3f064e4faaa072a48385ff8e [CANDIDATE]: Term 1 election: Requested vote from peers
I20260620 11:43:17.646288 16834 raft_consensus.cc:2781] T 00000000000000000000000000000000 P e0f9300b3f064e4faaa072a48385ff8e [term 1 FOLLOWER]: Leader election won for term 1
I20260620 11:43:17.646296 16829 sys_catalog.cc:564] T 00000000000000000000000000000000 P e0f9300b3f064e4faaa072a48385ff8e [sys.catalog]: configured and running, proceeding with master startup.
I20260620 11:43:17.646445 16834 raft_consensus.cc:686] T 00000000000000000000000000000000 P e0f9300b3f064e4faaa072a48385ff8e [term 1 LEADER]: Becoming Leader. State: Replica: e0f9300b3f064e4faaa072a48385ff8e, State: Running, Role: LEADER
I20260620 11:43:17.646544 16834 consensus_queue.cc:237] T 00000000000000000000000000000000 P e0f9300b3f064e4faaa072a48385ff8e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e0f9300b3f064e4faaa072a48385ff8e" member_type: VOTER last_known_addr { host: "127.6.234.254" port: 45627 } }
I20260620 11:43:17.646903 16836 sys_catalog.cc:455] T 00000000000000000000000000000000 P e0f9300b3f064e4faaa072a48385ff8e [sys.catalog]: SysCatalogTable state changed. Reason: New leader e0f9300b3f064e4faaa072a48385ff8e. Latest consensus state: current_term: 1 leader_uuid: "e0f9300b3f064e4faaa072a48385ff8e" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e0f9300b3f064e4faaa072a48385ff8e" member_type: VOTER last_known_addr { host: "127.6.234.254" port: 45627 } } }
I20260620 11:43:17.646996 16836 sys_catalog.cc:458] T 00000000000000000000000000000000 P e0f9300b3f064e4faaa072a48385ff8e [sys.catalog]: This master's current role is: LEADER
I20260620 11:43:17.646916 16835 sys_catalog.cc:455] T 00000000000000000000000000000000 P e0f9300b3f064e4faaa072a48385ff8e [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "e0f9300b3f064e4faaa072a48385ff8e" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e0f9300b3f064e4faaa072a48385ff8e" member_type: VOTER last_known_addr { host: "127.6.234.254" port: 45627 } } }
I20260620 11:43:17.647203 16835 sys_catalog.cc:458] T 00000000000000000000000000000000 P e0f9300b3f064e4faaa072a48385ff8e [sys.catalog]: This master's current role is: LEADER
I20260620 11:43:17.647212 16843 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260620 11:43:17.647969 16843 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260620 11:43:17.649399 16843 catalog_manager.cc:1347] Generated new cluster ID: 9490d8c5126a4898903486db292117b6
I20260620 11:43:17.649452 16843 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260620 11:43:17.672103 16843 catalog_manager.cc:1370] Generated new certificate authority record
I20260620 11:43:17.672317 16843 catalog_manager.cc:1504] Loading token signing keys...
I20260620 11:43:17.676780 16843 catalog_manager.cc:5868] T 00000000000000000000000000000000 P e0f9300b3f064e4faaa072a48385ff8e: Generated new TSK 0
I20260620 11:43:17.676951 16843 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260620 11:43:17.689203 7083 external_mini_cluster.cc:1326] Running /tmp/dist-test-task1JRg7N/build/release/bin/kudu
/tmp/dist-test-task1JRg7N/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task1JRg7N/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781955746769051-7083-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-task1JRg7N/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781955746769051-7083-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task1JRg7N/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781955746769051-7083-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-task1JRg7N/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781955746769051-7083-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.6.234.193:0
--local_ip_for_outbound_sockets=127.6.234.193
--webserver_interface=127.6.234.193
--webserver_port=0
--tserver_master_addrs=127.6.234.254:45627
--builtin_ntp_servers=127.6.234.212:34467
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_encryption=disabled
--rpc_authentication=disabled with env {}
W20260620 11:43:17.781208 16853 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 11:43:17.781406 16853 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 11:43:17.781428 16853 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 11:43:17.782639 16853 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 11:43:17.782687 16853 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 11:43:17.782713 16853 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.6.234.193
I20260620 11:43:17.784013 16853 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.6.234.212:34467
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task1JRg7N/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781955746769051-7083-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-task1JRg7N/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781955746769051-7083-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.6.234.193:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task1JRg7N/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781955746769051-7083-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.6.234.193
--webserver_port=0
--tserver_master_addrs=127.6.234.254:45627
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.16853
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.6.234.193
--log_dir=/tmp/dist-test-task1JRg7N/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781955746769051-7083-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 20 Jun 2026 10:32:37 UTC on ccycloud.quasar-bgyofv.root.comops.site
build id 4457173
I20260620 11:43:17.784247 16853 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 11:43:17.784535 16853 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 11:43:17.789377 16859 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 11:43:17.789386 16861 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 11:43:17.789397 16858 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 11:43:17.789685 16853 server_base.cc:1034] running on GCE node
I20260620 11:43:17.789913 16853 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 11:43:17.790148 16853 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 11:43:17.791307 16853 hybrid_clock.cc:648] HybridClock initialized: now 1781955797791297 us; error 31 us; skew 500 ppm
I20260620 11:43:17.792483 16853 webserver.cc:460] Webserver started at http://127.6.234.193:43565/ using document root <none> and password file <none>
I20260620 11:43:17.792728 16853 fs_manager.cc:359] Metadata directory not provided
I20260620 11:43:17.792800 16853 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 11:43:17.792953 16853 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 11:43:17.794070 16853 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task1JRg7N/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781955746769051-7083-0/minicluster-data/ts-0/data/instance:
uuid: "132eadb4eaa24cf59128ac59add220d4"
format_stamp: "Formatted at 2026-06-20 11:43:17 on dist-test-slave-cjc6"
server_key: "449e4dc0ccf03fba8f0a9199228ba330"
server_key_iv: "8ddfed54e426f77fe346744e94bebcb0"
server_key_version: "encryptionkey@0"
I20260620 11:43:17.794440 16853 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task1JRg7N/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781955746769051-7083-0/minicluster-data/ts-0/wal/instance:
uuid: "132eadb4eaa24cf59128ac59add220d4"
format_stamp: "Formatted at 2026-06-20 11:43:17 on dist-test-slave-cjc6"
server_key: "449e4dc0ccf03fba8f0a9199228ba330"
server_key_iv: "8ddfed54e426f77fe346744e94bebcb0"
server_key_version: "encryptionkey@0"
I20260620 11:43:17.795717 16853 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.001s sys 0.000s
I20260620 11:43:17.796399 16867 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 11:43:17.796589 16853 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260620 11:43:17.796695 16853 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task1JRg7N/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781955746769051-7083-0/minicluster-data/ts-0/data,/tmp/dist-test-task1JRg7N/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781955746769051-7083-0/minicluster-data/ts-0/wal
uuid: "132eadb4eaa24cf59128ac59add220d4"
format_stamp: "Formatted at 2026-06-20 11:43:17 on dist-test-slave-cjc6"
server_key: "449e4dc0ccf03fba8f0a9199228ba330"
server_key_iv: "8ddfed54e426f77fe346744e94bebcb0"
server_key_version: "encryptionkey@0"
I20260620 11:43:17.796772 16853 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task1JRg7N/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781955746769051-7083-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-task1JRg7N/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781955746769051-7083-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-task1JRg7N/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781955746769051-7083-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 11:43:17.799995 16853 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 11:43:17.800246 16853 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 11:43:17.800387 16853 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 11:43:17.800590 16853 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 11:43:17.800886 16853 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 11:43:17.800940 16853 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 11:43:17.800979 16853 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 11:43:17.801007 16853 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 11:43:17.806571 16853 rpc_server.cc:307] RPC server started. Bound to: 127.6.234.193:33991
I20260620 11:43:17.806656 16980 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.6.234.193:33991 every 8 connection(s)
I20260620 11:43:17.806948 16853 server_base.cc:1166] Dumped server information to /tmp/dist-test-task1JRg7N/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781955746769051-7083-0/minicluster-data/ts-0/data/info.pb
I20260620 11:43:17.810153 16832 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 11:43:17.807685 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.6.234.193:35279 (local address 127.6.234.254:45627)
0620 11:43:17.807756 (+ 71us) server_negotiation.cc:206] Beginning negotiation
0620 11:43:17.807759 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0620 11:43:17.807965 (+ 206us) server_negotiation.cc:407] Connection header received
0620 11:43:17.808480 (+ 515us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 11:43:17.808482 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 11:43:17.808490 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 11:43:17.808542 (+ 52us) server_negotiation.cc:226] Negotiated authn=SASL
0620 11:43:17.809882 (+ 1340us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 11:43:17.809884 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 11:43:17.809884 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 11:43:17.809890 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0620 11:43:17.809906 (+ 16us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 11:43:17.809911 (+ 5us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 11:43:17.809978 (+ 67us) server_negotiation.cc:1035] Waiting for connection context
0620 11:43:17.810063 (+ 85us) server_negotiation.cc:299] Negotiation successful
0620 11:43:17.810078 (+ 15us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":43}
I20260620 11:43:17.810348 16983 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 11:43:17.807675 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.234.254:45627 (local address 127.6.234.193:35279)
0620 11:43:17.807846 (+ 171us) negotiation.cc:107] Waiting for socket to connect
0620 11:43:17.807879 (+ 33us) client_negotiation.cc:174] Beginning negotiation
0620 11:43:17.808372 (+ 493us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 11:43:17.808625 (+ 253us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 11:43:17.808629 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 11:43:17.808640 (+ 11us) client_negotiation.cc:189] Negotiated authn=SASL
0620 11:43:17.809672 (+ 1032us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0620 11:43:17.809673 (+ 1us) client_negotiation.cc:656] calling sasl_client_start()
0620 11:43:17.809746 (+ 73us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0620 11:43:17.809784 (+ 38us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 11:43:17.809978 (+ 194us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 11:43:17.809984 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 11:43:17.809987 (+ 3us) client_negotiation.cc:769] Sending connection context
0620 11:43:17.810073 (+ 86us) client_negotiation.cc:240] Negotiation successful
0620 11:43:17.810135 (+ 62us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":102,"thread_start_us":66,"threads_started":1}
I20260620 11:43:17.810781 16981 heartbeater.cc:347] Connected to a master server at 127.6.234.254:45627
I20260620 11:43:17.810868 16981 heartbeater.cc:464] Registering TS with master...
I20260620 11:43:17.810984 16981 heartbeater.cc:511] Master 127.6.234.254:45627 requested a full tablet report, sending...
I20260620 11:43:17.811434 16793 ts_manager.cc:194] Registered new tserver with Master: 132eadb4eaa24cf59128ac59add220d4 (127.6.234.193:33991)
I20260620 11:43:17.814459 7083 external_mini_cluster.cc:1388] Started /tmp/dist-test-task1JRg7N/build/release/bin/kudu as pid 16853
I20260620 11:43:17.814585 7083 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task1JRg7N/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781955746769051-7083-0/minicluster-data/ts-0/wal/instance
I20260620 11:43:17.814766 7083 external_mini_cluster.cc:1428] Setting key 6eb467eae6da1590a520bbb308a1891a
I20260620 11:43:17.815724 7083 external_mini_cluster.cc:1326] Running /tmp/dist-test-task1JRg7N/build/release/bin/kudu
/tmp/dist-test-task1JRg7N/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task1JRg7N/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781955746769051-7083-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-task1JRg7N/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781955746769051-7083-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task1JRg7N/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781955746769051-7083-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-task1JRg7N/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781955746769051-7083-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.6.234.194:0
--local_ip_for_outbound_sockets=127.6.234.194
--webserver_interface=127.6.234.194
--webserver_port=0
--tserver_master_addrs=127.6.234.254:45627
--builtin_ntp_servers=127.6.234.212:34467
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_encryption=disabled
--rpc_authentication=disabled with env {}
W20260620 11:43:17.897353 16984 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 11:43:17.897603 16984 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 11:43:17.897642 16984 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 11:43:17.898849 16984 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 11:43:17.898914 16984 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 11:43:17.898960 16984 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.6.234.194
I20260620 11:43:17.900379 16984 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.6.234.212:34467
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task1JRg7N/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781955746769051-7083-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-task1JRg7N/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781955746769051-7083-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.6.234.194:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task1JRg7N/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781955746769051-7083-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.6.234.194
--webserver_port=0
--tserver_master_addrs=127.6.234.254:45627
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.16984
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.6.234.194
--log_dir=/tmp/dist-test-task1JRg7N/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781955746769051-7083-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 20 Jun 2026 10:32:37 UTC on ccycloud.quasar-bgyofv.root.comops.site
build id 4457173
I20260620 11:43:17.900628 16984 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 11:43:17.900909 16984 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260620 11:43:17.905956 16984 server_base.cc:1034] running on GCE node
W20260620 11:43:17.905893 16989 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 11:43:17.905892 16992 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 11:43:17.905917 16990 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 11:43:17.906338 16984 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 11:43:17.906572 16984 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 11:43:17.907709 16984 hybrid_clock.cc:648] HybridClock initialized: now 1781955797907689 us; error 34 us; skew 500 ppm
I20260620 11:43:17.908958 16984 webserver.cc:460] Webserver started at http://127.6.234.194:37923/ using document root <none> and password file <none>
I20260620 11:43:17.909204 16984 fs_manager.cc:359] Metadata directory not provided
I20260620 11:43:17.909251 16984 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 11:43:17.909410 16984 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 11:43:17.910496 16984 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task1JRg7N/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781955746769051-7083-0/minicluster-data/ts-1/data/instance:
uuid: "ec542bd99e154a81a446dc009df228eb"
format_stamp: "Formatted at 2026-06-20 11:43:17 on dist-test-slave-cjc6"
server_key: "43aad1a5b40ebc9b567e7c6df9ec1a9e"
server_key_iv: "e7a7c996badd7c5d9b7308887c33fca0"
server_key_version: "encryptionkey@0"
I20260620 11:43:17.910843 16984 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task1JRg7N/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781955746769051-7083-0/minicluster-data/ts-1/wal/instance:
uuid: "ec542bd99e154a81a446dc009df228eb"
format_stamp: "Formatted at 2026-06-20 11:43:17 on dist-test-slave-cjc6"
server_key: "43aad1a5b40ebc9b567e7c6df9ec1a9e"
server_key_iv: "e7a7c996badd7c5d9b7308887c33fca0"
server_key_version: "encryptionkey@0"
I20260620 11:43:17.912065 16984 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.002s
I20260620 11:43:17.912739 16998 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 11:43:17.912912 16984 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.001s
I20260620 11:43:17.912976 16984 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task1JRg7N/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781955746769051-7083-0/minicluster-data/ts-1/data,/tmp/dist-test-task1JRg7N/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781955746769051-7083-0/minicluster-data/ts-1/wal
uuid: "ec542bd99e154a81a446dc009df228eb"
format_stamp: "Formatted at 2026-06-20 11:43:17 on dist-test-slave-cjc6"
server_key: "43aad1a5b40ebc9b567e7c6df9ec1a9e"
server_key_iv: "e7a7c996badd7c5d9b7308887c33fca0"
server_key_version: "encryptionkey@0"
I20260620 11:43:17.913055 16984 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task1JRg7N/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781955746769051-7083-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-task1JRg7N/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781955746769051-7083-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-task1JRg7N/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781955746769051-7083-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 11:43:17.916308 16984 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 11:43:17.916543 16984 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 11:43:17.916677 16984 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 11:43:17.916877 16984 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 11:43:17.917155 16984 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 11:43:17.917187 16984 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 11:43:17.917238 16984 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 11:43:17.917266 16984 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 11:43:17.922822 16984 rpc_server.cc:307] RPC server started. Bound to: 127.6.234.194:43521
I20260620 11:43:17.922859 17111 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.6.234.194:43521 every 8 connection(s)
I20260620 11:43:17.923164 16984 server_base.cc:1166] Dumped server information to /tmp/dist-test-task1JRg7N/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781955746769051-7083-0/minicluster-data/ts-1/data/info.pb
I20260620 11:43:17.926155 16832 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 11:43:17.923867 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.6.234.194:52051 (local address 127.6.234.254:45627)
0620 11:43:17.923932 (+ 65us) server_negotiation.cc:206] Beginning negotiation
0620 11:43:17.923934 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 11:43:17.924152 (+ 218us) server_negotiation.cc:407] Connection header received
0620 11:43:17.924622 (+ 470us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 11:43:17.924623 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 11:43:17.924630 (+ 7us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 11:43:17.924679 (+ 49us) server_negotiation.cc:226] Negotiated authn=SASL
0620 11:43:17.925885 (+ 1206us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 11:43:17.925887 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 11:43:17.925887 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 11:43:17.925891 (+ 4us) server_negotiation.cc:922] calling sasl_server_start()
0620 11:43:17.925906 (+ 15us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 11:43:17.925910 (+ 4us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 11:43:17.925967 (+ 57us) server_negotiation.cc:1035] Waiting for connection context
0620 11:43:17.926066 (+ 99us) server_negotiation.cc:299] Negotiation successful
0620 11:43:17.926080 (+ 14us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":42}
I20260620 11:43:17.926337 17114 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 11:43:17.923861 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.234.254:45627 (local address 127.6.234.194:52051)
0620 11:43:17.924037 (+ 176us) negotiation.cc:107] Waiting for socket to connect
0620 11:43:17.924067 (+ 30us) client_negotiation.cc:174] Beginning negotiation
0620 11:43:17.924519 (+ 452us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 11:43:17.924746 (+ 227us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 11:43:17.924750 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 11:43:17.924758 (+ 8us) client_negotiation.cc:189] Negotiated authn=SASL
0620 11:43:17.925672 (+ 914us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0620 11:43:17.925672 (+ 0us) client_negotiation.cc:656] calling sasl_client_start()
0620 11:43:17.925760 (+ 88us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0620 11:43:17.925796 (+ 36us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 11:43:17.925968 (+ 172us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 11:43:17.925973 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 11:43:17.925975 (+ 2us) client_negotiation.cc:769] Sending connection context
0620 11:43:17.926073 (+ 98us) client_negotiation.cc:240] Negotiation successful
0620 11:43:17.926131 (+ 58us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":123,"thread_start_us":79,"threads_started":1}
I20260620 11:43:17.926779 17112 heartbeater.cc:347] Connected to a master server at 127.6.234.254:45627
I20260620 11:43:17.926848 17112 heartbeater.cc:464] Registering TS with master...
I20260620 11:43:17.927006 17112 heartbeater.cc:511] Master 127.6.234.254:45627 requested a full tablet report, sending...
I20260620 11:43:17.927351 16793 ts_manager.cc:194] Registered new tserver with Master: ec542bd99e154a81a446dc009df228eb (127.6.234.194:43521)
I20260620 11:43:17.930140 7083 external_mini_cluster.cc:1388] Started /tmp/dist-test-task1JRg7N/build/release/bin/kudu as pid 16984
I20260620 11:43:17.930250 7083 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task1JRg7N/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781955746769051-7083-0/minicluster-data/ts-1/wal/instance
I20260620 11:43:17.930465 7083 external_mini_cluster.cc:1428] Setting key 6980fb8f9e2496b17c545647d3c630b4
I20260620 11:43:17.931391 7083 external_mini_cluster.cc:1326] Running /tmp/dist-test-task1JRg7N/build/release/bin/kudu
/tmp/dist-test-task1JRg7N/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task1JRg7N/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781955746769051-7083-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-task1JRg7N/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781955746769051-7083-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task1JRg7N/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781955746769051-7083-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-task1JRg7N/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781955746769051-7083-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.6.234.195:0
--local_ip_for_outbound_sockets=127.6.234.195
--webserver_interface=127.6.234.195
--webserver_port=0
--tserver_master_addrs=127.6.234.254:45627
--builtin_ntp_servers=127.6.234.212:34467
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_encryption=disabled
--rpc_authentication=disabled with env {}
W20260620 11:43:18.011874 17115 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 11:43:18.012104 17115 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 11:43:18.012141 17115 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 11:43:18.013278 17115 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 11:43:18.013346 17115 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 11:43:18.013387 17115 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.6.234.195
I20260620 11:43:18.014714 17115 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.6.234.212:34467
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task1JRg7N/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781955746769051-7083-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-task1JRg7N/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781955746769051-7083-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.6.234.195:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task1JRg7N/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781955746769051-7083-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.6.234.195
--webserver_port=0
--tserver_master_addrs=127.6.234.254:45627
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.17115
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.6.234.195
--log_dir=/tmp/dist-test-task1JRg7N/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781955746769051-7083-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 20 Jun 2026 10:32:37 UTC on ccycloud.quasar-bgyofv.root.comops.site
build id 4457173
I20260620 11:43:18.014988 17115 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 11:43:18.015262 17115 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 11:43:18.020334 17120 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 11:43:18.020391 17121 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 11:43:18.020512 17115 server_base.cc:1034] running on GCE node
W20260620 11:43:18.020634 17123 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 11:43:18.020841 17115 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 11:43:18.021093 17115 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 11:43:18.022282 17115 hybrid_clock.cc:648] HybridClock initialized: now 1781955798022282 us; error 43 us; skew 500 ppm
I20260620 11:43:18.023592 17115 webserver.cc:460] Webserver started at http://127.6.234.195:46391/ using document root <none> and password file <none>
I20260620 11:43:18.023835 17115 fs_manager.cc:359] Metadata directory not provided
I20260620 11:43:18.023905 17115 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 11:43:18.024047 17115 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 11:43:18.025063 17115 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task1JRg7N/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781955746769051-7083-0/minicluster-data/ts-2/data/instance:
uuid: "db6fc2a5d82f4f34a93dbc7819fefe81"
format_stamp: "Formatted at 2026-06-20 11:43:18 on dist-test-slave-cjc6"
server_key: "8c6ed1dda982c282c29aaa5714c60b43"
server_key_iv: "93960111ecb7726565cb8c51fbf2b77a"
server_key_version: "encryptionkey@0"
I20260620 11:43:18.025430 17115 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task1JRg7N/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781955746769051-7083-0/minicluster-data/ts-2/wal/instance:
uuid: "db6fc2a5d82f4f34a93dbc7819fefe81"
format_stamp: "Formatted at 2026-06-20 11:43:18 on dist-test-slave-cjc6"
server_key: "8c6ed1dda982c282c29aaa5714c60b43"
server_key_iv: "93960111ecb7726565cb8c51fbf2b77a"
server_key_version: "encryptionkey@0"
I20260620 11:43:18.026873 17115 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260620 11:43:18.027614 17129 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 11:43:18.027784 17115 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260620 11:43:18.027882 17115 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task1JRg7N/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781955746769051-7083-0/minicluster-data/ts-2/data,/tmp/dist-test-task1JRg7N/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781955746769051-7083-0/minicluster-data/ts-2/wal
uuid: "db6fc2a5d82f4f34a93dbc7819fefe81"
format_stamp: "Formatted at 2026-06-20 11:43:18 on dist-test-slave-cjc6"
server_key: "8c6ed1dda982c282c29aaa5714c60b43"
server_key_iv: "93960111ecb7726565cb8c51fbf2b77a"
server_key_version: "encryptionkey@0"
I20260620 11:43:18.027969 17115 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task1JRg7N/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781955746769051-7083-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-task1JRg7N/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781955746769051-7083-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-task1JRg7N/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781955746769051-7083-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 11:43:18.031327 17115 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 11:43:18.031587 17115 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 11:43:18.031725 17115 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 11:43:18.031945 17115 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 11:43:18.032262 17115 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 11:43:18.032316 17115 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 11:43:18.032356 17115 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 11:43:18.032383 17115 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 11:43:18.038241 17115 rpc_server.cc:307] RPC server started. Bound to: 127.6.234.195:42401
I20260620 11:43:18.038276 17242 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.6.234.195:42401 every 8 connection(s)
I20260620 11:43:18.038617 17115 server_base.cc:1166] Dumped server information to /tmp/dist-test-task1JRg7N/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781955746769051-7083-0/minicluster-data/ts-2/data/info.pb
I20260620 11:43:18.041456 16832 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 11:43:18.039314 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.6.234.195:35819 (local address 127.6.234.254:45627)
0620 11:43:18.039376 (+ 62us) server_negotiation.cc:206] Beginning negotiation
0620 11:43:18.039387 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0620 11:43:18.039618 (+ 231us) server_negotiation.cc:407] Connection header received
0620 11:43:18.040040 (+ 422us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 11:43:18.040041 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 11:43:18.040045 (+ 4us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 11:43:18.040106 (+ 61us) server_negotiation.cc:226] Negotiated authn=SASL
0620 11:43:18.041260 (+ 1154us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 11:43:18.041261 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 11:43:18.041261 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 11:43:18.041265 (+ 4us) server_negotiation.cc:922] calling sasl_server_start()
0620 11:43:18.041277 (+ 12us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 11:43:18.041281 (+ 4us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 11:43:18.041327 (+ 46us) server_negotiation.cc:1035] Waiting for connection context
0620 11:43:18.041391 (+ 64us) server_negotiation.cc:299] Negotiation successful
0620 11:43:18.041404 (+ 13us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":35}
I20260620 11:43:18.041625 17245 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 11:43:18.039309 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.234.254:45627 (local address 127.6.234.195:35819)
0620 11:43:18.039495 (+ 186us) negotiation.cc:107] Waiting for socket to connect
0620 11:43:18.039534 (+ 39us) client_negotiation.cc:174] Beginning negotiation
0620 11:43:18.039960 (+ 426us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 11:43:18.040155 (+ 195us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 11:43:18.040158 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 11:43:18.040166 (+ 8us) client_negotiation.cc:189] Negotiated authn=SASL
0620 11:43:18.041117 (+ 951us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0620 11:43:18.041117 (+ 0us) client_negotiation.cc:656] calling sasl_client_start()
0620 11:43:18.041146 (+ 29us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0620 11:43:18.041174 (+ 28us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 11:43:18.041328 (+ 154us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 11:43:18.041330 (+ 2us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 11:43:18.041332 (+ 2us) client_negotiation.cc:769] Sending connection context
0620 11:43:18.041384 (+ 52us) client_negotiation.cc:240] Negotiation successful
0620 11:43:18.041430 (+ 46us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":123,"thread_start_us":96,"threads_started":1}
I20260620 11:43:18.042181 17243 heartbeater.cc:347] Connected to a master server at 127.6.234.254:45627
I20260620 11:43:18.042284 17243 heartbeater.cc:464] Registering TS with master...
I20260620 11:43:18.042428 17243 heartbeater.cc:511] Master 127.6.234.254:45627 requested a full tablet report, sending...
I20260620 11:43:18.042797 16793 ts_manager.cc:194] Registered new tserver with Master: db6fc2a5d82f4f34a93dbc7819fefe81 (127.6.234.195:42401)
I20260620 11:43:18.045878 7083 external_mini_cluster.cc:1388] Started /tmp/dist-test-task1JRg7N/build/release/bin/kudu as pid 17115
I20260620 11:43:18.045979 7083 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task1JRg7N/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781955746769051-7083-0/minicluster-data/ts-2/wal/instance
I20260620 11:43:18.046169 7083 external_mini_cluster.cc:1428] Setting key a644fbf783a8e8a8e8b0807d3eec2169
I20260620 11:43:18.047057 7083 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
W20260620 11:43:18.048771 16832 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 11:43:18.048413 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58588 (local address 127.6.234.254:45627)
0620 11:43:18.048480 (+ 67us) server_negotiation.cc:206] Beginning negotiation
0620 11:43:18.048483 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0620 11:43:18.048549 (+ 66us) server_negotiation.cc:407] Connection header received
0620 11:43:18.048589 (+ 40us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 11:43:18.048589 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 11:43:18.048594 (+ 5us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 11:43:18.048637 (+ 43us) server_negotiation.cc:226] Negotiated authn=SASL
0620 11:43:18.048727 (+ 90us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:58588: BlockingRecv error: recv got EOF from 127.0.0.1:58588 (error 108)
Metrics: {"server-negotiator.queue_time_us":37}
I20260620 11:43:18.048848 7083 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task1JRg7N/build/release/bin/kudu with pid 16853
I20260620 11:43:18.053510 7083 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task1JRg7N/build/release/bin/kudu with pid 16984
I20260620 11:43:18.057818 7083 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task1JRg7N/build/release/bin/kudu with pid 17115
I20260620 11:43:18.061872 7083 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task1JRg7N/build/release/bin/kudu with pid 16762
2026-06-20T11:43:18Z chronyd exiting
[ OK ] SecurityITest.TestRequireEncryptionInsecureCluster (552 ms)
[ RUN ] SecurityITest.TestRequireAuthenticationSecureCluster
Loading random data
Initializing database '/tmp/dist-test-task1JRg7N/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781955746769051-7083-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 20 11:43:18 dist-test-slave-cjc6 krb5kdc[17253](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 20 11:43:18 dist-test-slave-cjc6 krb5kdc[17253](info): set up 2 sockets
Jun 20 11:43:18 dist-test-slave-cjc6 krb5kdc[17253](info): commencing operation
krb5kdc: starting...
W20260620 11:43:19.107352 7083 mini_kdc.cc:121] Time spent starting KDC: real 1.026s user 0.000s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task1JRg7N/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781955746769051-7083-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task1JRg7N/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781955746769051-7083-0/krb5kdc/test-user.keytab.
Jun 20 11:43:19 dist-test-slave-cjc6 krb5kdc[17253](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781955799, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-20T11:43:19Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-20T11:43:19Z Disabled control of system clock
WARNING: no policy specified for kudu/127.6.234.254@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.6.234.254@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.6.234.254 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task1JRg7N/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781955746769051-7083-0/krb5kdc/kudu_127.6.234.254.keytab.
Entry for principal kudu/127.6.234.254 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task1JRg7N/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781955746769051-7083-0/krb5kdc/kudu_127.6.234.254.keytab.
I20260620 11:43:19.218448 7083 external_mini_cluster.cc:1326] Running /tmp/dist-test-task1JRg7N/build/release/bin/kudu
/tmp/dist-test-task1JRg7N/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task1JRg7N/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781955746769051-7083-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task1JRg7N/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781955746769051-7083-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task1JRg7N/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781955746769051-7083-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task1JRg7N/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781955746769051-7083-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.6.234.254:40311
--webserver_interface=127.6.234.254
--webserver_port=0
--keytab_file=/tmp/dist-test-task1JRg7N/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781955746769051-7083-0/krb5kdc/kudu_127.6.234.254.keytab
--principal=kudu/127.6.234.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.6.234.212:43989
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.6.234.254:40311
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task1JRg7N/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781955746769051-7083-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task1JRg7N/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781955746769051-7083-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task1JRg7N/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781955746769051-7083-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 11:43:19.297081 17266 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 11:43:19.297317 17266 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 11:43:19.297353 17266 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 11:43:19.298573 17266 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 11:43:19.298640 17266 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 11:43:19.298664 17266 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 11:43:19.298693 17266 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 11:43:19.298717 17266 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 11:43:19.300180 17266 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.6.234.212:43989
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task1JRg7N/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781955746769051-7083-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task1JRg7N/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781955746769051-7083-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.6.234.254:40311
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task1JRg7N/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781955746769051-7083-0/krb5kdc/kudu_127.6.234.254.keytab
--principal=kudu/127.6.234.254
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.6.234.254:40311
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task1JRg7N/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781955746769051-7083-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.6.234.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.17266
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task1JRg7N/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781955746769051-7083-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 20 Jun 2026 10:32:37 UTC on ccycloud.quasar-bgyofv.root.comops.site
build id 4457173
I20260620 11:43:19.300451 17266 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 11:43:19.300673 17266 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 11:43:19.305370 17271 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 11:43:19.305377 17274 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 11:43:19.305372 17272 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 11:43:19.305630 17266 server_base.cc:1034] running on GCE node
I20260620 11:43:19.305872 17266 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 11:43:19.306152 17266 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 11:43:19.307330 17266 hybrid_clock.cc:648] HybridClock initialized: now 1781955799307315 us; error 41 us; skew 500 ppm
Jun 20 11:43:19 dist-test-slave-cjc6 krb5kdc[17253](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781955799, etypes {rep=17 tkt=17 ses=17}, kudu/127.6.234.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 11:43:19.310647 17266 init.cc:377] Logged in from keytab as kudu/127.6.234.254@KRBTEST.COM (short username kudu)
I20260620 11:43:19.311493 17266 webserver.cc:460] Webserver started at http://127.6.234.254:41113/ using document root <none> and password file <none>
I20260620 11:43:19.311813 17266 fs_manager.cc:359] Metadata directory not provided
I20260620 11:43:19.311874 17266 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 11:43:19.312017 17266 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 11:43:19.313925 17266 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task1JRg7N/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781955746769051-7083-0/minicluster-data/master-0/data/instance:
uuid: "6a71c9c9868b487c85eb2b417342a378"
format_stamp: "Formatted at 2026-06-20 11:43:19 on dist-test-slave-cjc6"
server_key: "318b79c1a473db0b8897d396133641f4"
server_key_iv: "3b4b51afff620a5a594538b87bd1a383"
server_key_version: "encryptionkey@0"
I20260620 11:43:19.314342 17266 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task1JRg7N/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781955746769051-7083-0/minicluster-data/master-0/wal/instance:
uuid: "6a71c9c9868b487c85eb2b417342a378"
format_stamp: "Formatted at 2026-06-20 11:43:19 on dist-test-slave-cjc6"
server_key: "318b79c1a473db0b8897d396133641f4"
server_key_iv: "3b4b51afff620a5a594538b87bd1a383"
server_key_version: "encryptionkey@0"
I20260620 11:43:19.316211 17266 fs_manager.cc:691] Time spent creating directory manager: real 0.002s user 0.001s sys 0.001s
I20260620 11:43:19.317405 17281 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 11:43:19.317610 17266 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260620 11:43:19.317724 17266 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task1JRg7N/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781955746769051-7083-0/minicluster-data/master-0/data,/tmp/dist-test-task1JRg7N/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781955746769051-7083-0/minicluster-data/master-0/wal
uuid: "6a71c9c9868b487c85eb2b417342a378"
format_stamp: "Formatted at 2026-06-20 11:43:19 on dist-test-slave-cjc6"
server_key: "318b79c1a473db0b8897d396133641f4"
server_key_iv: "3b4b51afff620a5a594538b87bd1a383"
server_key_version: "encryptionkey@0"
I20260620 11:43:19.318034 17266 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task1JRg7N/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781955746769051-7083-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task1JRg7N/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781955746769051-7083-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task1JRg7N/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781955746769051-7083-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 11:43:19.349411 17266 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 11:43:19.352098 17266 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 11:43:19.352255 17266 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 11:43:19.356263 17266 rpc_server.cc:307] RPC server started. Bound to: 127.6.234.254:40311
I20260620 11:43:19.356297 17333 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.6.234.254:40311 every 8 connection(s)
I20260620 11:43:19.356684 17266 server_base.cc:1166] Dumped server information to /tmp/dist-test-task1JRg7N/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781955746769051-7083-0/minicluster-data/master-0/data/info.pb
I20260620 11:43:19.357677 17334 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 11:43:19.359969 17334 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 6a71c9c9868b487c85eb2b417342a378: Bootstrap starting.
I20260620 11:43:19.360551 17334 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 6a71c9c9868b487c85eb2b417342a378: Neither blocks nor log segments found. Creating new log.
I20260620 11:43:19.360836 17334 log.cc:824] T 00000000000000000000000000000000 P 6a71c9c9868b487c85eb2b417342a378: Log is configured to *not* fsync() on all Append() calls
I20260620 11:43:19.361501 17334 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 6a71c9c9868b487c85eb2b417342a378: No bootstrap required, opened a new log
I20260620 11:43:19.362685 7083 external_mini_cluster.cc:1388] Started /tmp/dist-test-task1JRg7N/build/release/bin/kudu as pid 17266
I20260620 11:43:19.362771 7083 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task1JRg7N/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781955746769051-7083-0/minicluster-data/master-0/wal/instance
I20260620 11:43:19.362990 7083 external_mini_cluster.cc:1428] Setting key 1ba153eb8e59f121a2bdf9bc391c6bde
I20260620 11:43:19.363744 17334 raft_consensus.cc:348] T 00000000000000000000000000000000 P 6a71c9c9868b487c85eb2b417342a378 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6a71c9c9868b487c85eb2b417342a378" member_type: VOTER last_known_addr { host: "127.6.234.254" port: 40311 } }
I20260620 11:43:19.363925 17334 raft_consensus.cc:374] T 00000000000000000000000000000000 P 6a71c9c9868b487c85eb2b417342a378 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 11:43:19.363961 17334 raft_consensus.cc:729] T 00000000000000000000000000000000 P 6a71c9c9868b487c85eb2b417342a378 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6a71c9c9868b487c85eb2b417342a378, State: Initialized, Role: FOLLOWER
I20260620 11:43:19.364091 17334 consensus_queue.cc:260] T 00000000000000000000000000000000 P 6a71c9c9868b487c85eb2b417342a378 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6a71c9c9868b487c85eb2b417342a378" member_type: VOTER last_known_addr { host: "127.6.234.254" port: 40311 } }
I20260620 11:43:19.364182 17334 raft_consensus.cc:388] T 00000000000000000000000000000000 P 6a71c9c9868b487c85eb2b417342a378 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260620 11:43:19.364212 17334 raft_consensus.cc:482] T 00000000000000000000000000000000 P 6a71c9c9868b487c85eb2b417342a378 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260620 11:43:19.364260 17334 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 6a71c9c9868b487c85eb2b417342a378 [term 0 FOLLOWER]: Advancing to term 1
I20260620 11:43:19.364900 17334 raft_consensus.cc:504] T 00000000000000000000000000000000 P 6a71c9c9868b487c85eb2b417342a378 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6a71c9c9868b487c85eb2b417342a378" member_type: VOTER last_known_addr { host: "127.6.234.254" port: 40311 } }
I20260620 11:43:19.365002 17334 leader_election.cc:302] T 00000000000000000000000000000000 P 6a71c9c9868b487c85eb2b417342a378 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 6a71c9c9868b487c85eb2b417342a378; no voters:
I20260620 11:43:19.365155 17334 leader_election.cc:288] T 00000000000000000000000000000000 P 6a71c9c9868b487c85eb2b417342a378 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260620 11:43:19.365255 17339 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 6a71c9c9868b487c85eb2b417342a378 [term 1 FOLLOWER]: Leader election won for term 1
I20260620 11:43:19.365435 17334 sys_catalog.cc:564] T 00000000000000000000000000000000 P 6a71c9c9868b487c85eb2b417342a378 [sys.catalog]: configured and running, proceeding with master startup.
I20260620 11:43:19.365449 17339 raft_consensus.cc:686] T 00000000000000000000000000000000 P 6a71c9c9868b487c85eb2b417342a378 [term 1 LEADER]: Becoming Leader. State: Replica: 6a71c9c9868b487c85eb2b417342a378, State: Running, Role: LEADER
I20260620 11:43:19.365648 17339 consensus_queue.cc:237] T 00000000000000000000000000000000 P 6a71c9c9868b487c85eb2b417342a378 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6a71c9c9868b487c85eb2b417342a378" member_type: VOTER last_known_addr { host: "127.6.234.254" port: 40311 } }
I20260620 11:43:19.366099 17340 sys_catalog.cc:455] T 00000000000000000000000000000000 P 6a71c9c9868b487c85eb2b417342a378 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "6a71c9c9868b487c85eb2b417342a378" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6a71c9c9868b487c85eb2b417342a378" member_type: VOTER last_known_addr { host: "127.6.234.254" port: 40311 } } }
I20260620 11:43:19.366195 17340 sys_catalog.cc:458] T 00000000000000000000000000000000 P 6a71c9c9868b487c85eb2b417342a378 [sys.catalog]: This master's current role is: LEADER
I20260620 11:43:19.366328 17340 sys_catalog.cc:455] T 00000000000000000000000000000000 P 6a71c9c9868b487c85eb2b417342a378 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 6a71c9c9868b487c85eb2b417342a378. Latest consensus state: current_term: 1 leader_uuid: "6a71c9c9868b487c85eb2b417342a378" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6a71c9c9868b487c85eb2b417342a378" member_type: VOTER last_known_addr { host: "127.6.234.254" port: 40311 } } }
I20260620 11:43:19.366376 17340 sys_catalog.cc:458] T 00000000000000000000000000000000 P 6a71c9c9868b487c85eb2b417342a378 [sys.catalog]: This master's current role is: LEADER
I20260620 11:43:19.366770 17348 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260620 11:43:19.367427 17348 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260620 11:43:19.368829 17348 catalog_manager.cc:1347] Generated new cluster ID: b5c05a522cd44db08ecaa11b5074c515
I20260620 11:43:19.368882 17348 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
Jun 20 11:43:19 dist-test-slave-cjc6 krb5kdc[17253](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781955799, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.6.234.254@KRBTEST.COM
I20260620 11:43:19.374650 17338 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 11:43:19.363559 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48992 (local address 127.6.234.254:40311)
0620 11:43:19.363751 (+ 192us) server_negotiation.cc:206] Beginning negotiation
0620 11:43:19.363761 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0620 11:43:19.363796 (+ 35us) server_negotiation.cc:407] Connection header received
0620 11:43:19.363903 (+ 107us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 11:43:19.363910 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 11:43:19.363968 (+ 58us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 11:43:19.364109 (+ 141us) server_negotiation.cc:226] Negotiated authn=SASL
0620 11:43:19.364791 (+ 682us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 11:43:19.366286 (+ 1495us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 11:43:19.367575 (+ 1289us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 11:43:19.368168 (+ 593us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 11:43:19.368300 (+ 132us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 11:43:19.370744 (+ 2444us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 11:43:19.370748 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 11:43:19.370748 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 11:43:19.370759 (+ 11us) server_negotiation.cc:922] calling sasl_server_start()
0620 11:43:19.372386 (+ 1627us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 11:43:19.372755 (+ 369us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 11:43:19.372759 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 11:43:19.372759 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 11:43:19.372894 (+ 135us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 11:43:19.373114 (+ 220us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 11:43:19.373115 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 11:43:19.373115 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 11:43:19.373462 (+ 347us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 11:43:19.374087 (+ 625us) server_negotiation.cc:1035] Waiting for connection context
0620 11:43:19.374320 (+ 233us) server_negotiation.cc:299] Negotiation successful
0620 11:43:19.374458 (+ 138us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":143,"thread_start_us":89,"threads_started":1}
I20260620 11:43:19.398182 17348 catalog_manager.cc:1370] Generated new certificate authority record
I20260620 11:43:19.398767 17348 catalog_manager.cc:1504] Loading token signing keys...
I20260620 11:43:19.414624 17348 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 6a71c9c9868b487c85eb2b417342a378: Generated new TSK 0
I20260620 11:43:19.414839 17348 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.6.234.193@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.6.234.193@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.6.234.193 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task1JRg7N/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781955746769051-7083-0/krb5kdc/kudu_127.6.234.193.keytab.
Entry for principal kudu/127.6.234.193 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task1JRg7N/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781955746769051-7083-0/krb5kdc/kudu_127.6.234.193.keytab.
I20260620 11:43:19.453037 7083 external_mini_cluster.cc:1326] Running /tmp/dist-test-task1JRg7N/build/release/bin/kudu
/tmp/dist-test-task1JRg7N/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task1JRg7N/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781955746769051-7083-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-task1JRg7N/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781955746769051-7083-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task1JRg7N/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781955746769051-7083-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-task1JRg7N/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781955746769051-7083-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.6.234.193:0
--local_ip_for_outbound_sockets=127.6.234.193
--webserver_interface=127.6.234.193
--webserver_port=0
--tserver_master_addrs=127.6.234.254:40311
--keytab_file=/tmp/dist-test-task1JRg7N/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781955746769051-7083-0/krb5kdc/kudu_127.6.234.193.keytab
--principal=kudu/127.6.234.193
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.6.234.212:43989
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task1JRg7N/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781955746769051-7083-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task1JRg7N/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781955746769051-7083-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task1JRg7N/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781955746769051-7083-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 11:43:19.547847 17360 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 11:43:19.548112 17360 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 11:43:19.548137 17360 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 11:43:19.549263 17360 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 11:43:19.549302 17360 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 11:43:19.549355 17360 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.6.234.193
I20260620 11:43:19.550792 17360 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.6.234.212:43989
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task1JRg7N/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781955746769051-7083-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-task1JRg7N/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781955746769051-7083-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task1JRg7N/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781955746769051-7083-0/krb5kdc/kudu_127.6.234.193.keytab
--principal=kudu/127.6.234.193
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.6.234.193:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task1JRg7N/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781955746769051-7083-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.6.234.193
--webserver_port=0
--tserver_master_addrs=127.6.234.254:40311
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.17360
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.6.234.193
--log_dir=/tmp/dist-test-task1JRg7N/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781955746769051-7083-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 20 Jun 2026 10:32:37 UTC on ccycloud.quasar-bgyofv.root.comops.site
build id 4457173
I20260620 11:43:19.551024 17360 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 11:43:19.551270 17360 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260620 11:43:19.556355 17360 server_base.cc:1034] running on GCE node
W20260620 11:43:19.556339 17366 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 11:43:19.556339 17365 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 11:43:19.556329 17368 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 11:43:19.556707 17360 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 11:43:19.556944 17360 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 11:43:19.558112 17360 hybrid_clock.cc:648] HybridClock initialized: now 1781955799558098 us; error 45 us; skew 500 ppm
Jun 20 11:43:19 dist-test-slave-cjc6 krb5kdc[17253](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781955799, etypes {rep=17 tkt=17 ses=17}, kudu/127.6.234.193@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 11:43:19.560588 17360 init.cc:377] Logged in from keytab as kudu/127.6.234.193@KRBTEST.COM (short username kudu)
I20260620 11:43:19.560989 17360 webserver.cc:460] Webserver started at http://127.6.234.193:44081/ using document root <none> and password file <none>
I20260620 11:43:19.561215 17360 fs_manager.cc:359] Metadata directory not provided
I20260620 11:43:19.561283 17360 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 11:43:19.561432 17360 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 11:43:19.562502 17360 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task1JRg7N/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781955746769051-7083-0/minicluster-data/ts-0/data/instance:
uuid: "b36a5c84137e48baaa4d49f2b9dcf6e8"
format_stamp: "Formatted at 2026-06-20 11:43:19 on dist-test-slave-cjc6"
server_key: "399b5ce26320727210b3dedd9d9a6c10"
server_key_iv: "c17cd50c124b2a6eadf2548165a13863"
server_key_version: "encryptionkey@0"
I20260620 11:43:19.562858 17360 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task1JRg7N/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781955746769051-7083-0/minicluster-data/ts-0/wal/instance:
uuid: "b36a5c84137e48baaa4d49f2b9dcf6e8"
format_stamp: "Formatted at 2026-06-20 11:43:19 on dist-test-slave-cjc6"
server_key: "399b5ce26320727210b3dedd9d9a6c10"
server_key_iv: "c17cd50c124b2a6eadf2548165a13863"
server_key_version: "encryptionkey@0"
I20260620 11:43:19.564117 17360 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.002s
I20260620 11:43:19.564862 17375 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 11:43:19.565056 17360 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.001s
I20260620 11:43:19.565161 17360 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task1JRg7N/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781955746769051-7083-0/minicluster-data/ts-0/data,/tmp/dist-test-task1JRg7N/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781955746769051-7083-0/minicluster-data/ts-0/wal
uuid: "b36a5c84137e48baaa4d49f2b9dcf6e8"
format_stamp: "Formatted at 2026-06-20 11:43:19 on dist-test-slave-cjc6"
server_key: "399b5ce26320727210b3dedd9d9a6c10"
server_key_iv: "c17cd50c124b2a6eadf2548165a13863"
server_key_version: "encryptionkey@0"
I20260620 11:43:19.565233 17360 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task1JRg7N/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781955746769051-7083-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-task1JRg7N/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781955746769051-7083-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-task1JRg7N/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781955746769051-7083-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 11:43:19.591990 17360 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 11:43:19.593786 17360 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 11:43:19.593945 17360 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 11:43:19.594172 17360 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 11:43:19.594480 17360 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 11:43:19.594542 17360 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 11:43:19.594594 17360 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 11:43:19.594625 17360 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 11:43:19.600304 17360 rpc_server.cc:307] RPC server started. Bound to: 127.6.234.193:42267
I20260620 11:43:19.600349 17488 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.6.234.193:42267 every 8 connection(s)
I20260620 11:43:19.600657 17360 server_base.cc:1166] Dumped server information to /tmp/dist-test-task1JRg7N/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781955746769051-7083-0/minicluster-data/ts-0/data/info.pb
Jun 20 11:43:19 dist-test-slave-cjc6 krb5kdc[17253](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781955799, etypes {rep=17 tkt=17 ses=17}, kudu/127.6.234.193@KRBTEST.COM for kudu/127.6.234.254@KRBTEST.COM
I20260620 11:43:19.608925 7083 external_mini_cluster.cc:1388] Started /tmp/dist-test-task1JRg7N/build/release/bin/kudu as pid 17360
I20260620 11:43:19.609028 7083 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task1JRg7N/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781955746769051-7083-0/minicluster-data/ts-0/wal/instance
I20260620 11:43:19.609248 7083 external_mini_cluster.cc:1428] Setting key 13b176c8490a58583a99f4f7b7b0463a
I20260620 11:43:19.609597 17338 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 11:43:19.601313 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.6.234.193:38437 (local address 127.6.234.254:40311)
0620 11:43:19.601398 (+ 85us) server_negotiation.cc:206] Beginning negotiation
0620 11:43:19.601400 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 11:43:19.601582 (+ 182us) server_negotiation.cc:407] Connection header received
0620 11:43:19.602056 (+ 474us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 11:43:19.602057 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 11:43:19.602073 (+ 16us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 11:43:19.602125 (+ 52us) server_negotiation.cc:226] Negotiated authn=SASL
0620 11:43:19.603020 (+ 895us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 11:43:19.603516 (+ 496us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 11:43:19.604458 (+ 942us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 11:43:19.604673 (+ 215us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 11:43:19.604738 (+ 65us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 11:43:19.606829 (+ 2091us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 11:43:19.606831 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 11:43:19.606831 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 11:43:19.606837 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0620 11:43:19.608284 (+ 1447us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 11:43:19.608614 (+ 330us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 11:43:19.608615 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 11:43:19.608616 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 11:43:19.608663 (+ 47us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 11:43:19.608895 (+ 232us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 11:43:19.608896 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 11:43:19.608897 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 11:43:19.609044 (+ 147us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 11:43:19.609109 (+ 65us) server_negotiation.cc:1035] Waiting for connection context
0620 11:43:19.609408 (+ 299us) server_negotiation.cc:299] Negotiation successful
0620 11:43:19.609493 (+ 85us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":54}
I20260620 11:43:19.609845 17491 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 11:43:19.601282 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.234.254:40311 (local address 127.6.234.193:38437)
0620 11:43:19.601461 (+ 179us) negotiation.cc:107] Waiting for socket to connect
0620 11:43:19.601496 (+ 35us) client_negotiation.cc:174] Beginning negotiation
0620 11:43:19.601974 (+ 478us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 11:43:19.602177 (+ 203us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 11:43:19.602180 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 11:43:19.602737 (+ 557us) client_negotiation.cc:189] Negotiated authn=SASL
0620 11:43:19.602920 (+ 183us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 11:43:19.602925 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 11:43:19.603604 (+ 679us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 11:43:19.603605 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 11:43:19.604366 (+ 761us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 11:43:19.604368 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 11:43:19.604753 (+ 385us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 11:43:19.604753 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 11:43:19.604848 (+ 95us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 11:43:19.605260 (+ 412us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 11:43:19.605267 (+ 7us) client_negotiation.cc:656] calling sasl_client_start()
0620 11:43:19.606735 (+ 1468us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 11:43:19.608350 (+ 1615us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 11:43:19.608355 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 11:43:19.608356 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0620 11:43:19.608551 (+ 195us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 11:43:19.608727 (+ 176us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 11:43:19.608728 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 11:43:19.608728 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0620 11:43:19.608829 (+ 101us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 11:43:19.609139 (+ 310us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 11:43:19.609144 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 11:43:19.609252 (+ 108us) client_negotiation.cc:769] Sending connection context
0620 11:43:19.609371 (+ 119us) client_negotiation.cc:240] Negotiation successful
0620 11:43:19.609519 (+ 148us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":143,"thread_start_us":101,"threads_started":1}
I20260620 11:43:19.611469 17489 heartbeater.cc:347] Connected to a master server at 127.6.234.254:40311
I20260620 11:43:19.611559 17489 heartbeater.cc:464] Registering TS with master...
I20260620 11:43:19.611739 17489 heartbeater.cc:511] Master 127.6.234.254:40311 requested a full tablet report, sending...
I20260620 11:43:19.612263 17298 ts_manager.cc:194] Registered new tserver with Master: b36a5c84137e48baaa4d49f2b9dcf6e8 (127.6.234.193:42267)
I20260620 11:43:19.613204 17298 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.6.234.193@KRBTEST.COM'} at 127.6.234.193:38437
WARNING: no policy specified for kudu/127.6.234.194@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.6.234.194@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.6.234.194 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task1JRg7N/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781955746769051-7083-0/krb5kdc/kudu_127.6.234.194.keytab.
Entry for principal kudu/127.6.234.194 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task1JRg7N/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781955746769051-7083-0/krb5kdc/kudu_127.6.234.194.keytab.
I20260620 11:43:19.634138 7083 external_mini_cluster.cc:1326] Running /tmp/dist-test-task1JRg7N/build/release/bin/kudu
/tmp/dist-test-task1JRg7N/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task1JRg7N/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781955746769051-7083-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-task1JRg7N/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781955746769051-7083-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task1JRg7N/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781955746769051-7083-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-task1JRg7N/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781955746769051-7083-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.6.234.194:0
--local_ip_for_outbound_sockets=127.6.234.194
--webserver_interface=127.6.234.194
--webserver_port=0
--tserver_master_addrs=127.6.234.254:40311
--keytab_file=/tmp/dist-test-task1JRg7N/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781955746769051-7083-0/krb5kdc/kudu_127.6.234.194.keytab
--principal=kudu/127.6.234.194
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.6.234.212:43989
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task1JRg7N/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781955746769051-7083-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task1JRg7N/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781955746769051-7083-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task1JRg7N/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781955746769051-7083-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 11:43:19.712589 17494 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 11:43:19.712816 17494 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 11:43:19.712857 17494 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 11:43:19.714058 17494 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 11:43:19.714124 17494 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 11:43:19.714170 17494 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.6.234.194
I20260620 11:43:19.715616 17494 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.6.234.212:43989
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task1JRg7N/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781955746769051-7083-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-task1JRg7N/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781955746769051-7083-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task1JRg7N/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781955746769051-7083-0/krb5kdc/kudu_127.6.234.194.keytab
--principal=kudu/127.6.234.194
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.6.234.194:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task1JRg7N/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781955746769051-7083-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.6.234.194
--webserver_port=0
--tserver_master_addrs=127.6.234.254:40311
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.17494
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.6.234.194
--log_dir=/tmp/dist-test-task1JRg7N/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781955746769051-7083-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 20 Jun 2026 10:32:37 UTC on ccycloud.quasar-bgyofv.root.comops.site
build id 4457173
I20260620 11:43:19.715865 17494 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 11:43:19.716084 17494 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 11:43:19.721069 17500 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 11:43:19.721127 17502 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 11:43:19.721256 17494 server_base.cc:1034] running on GCE node
W20260620 11:43:19.721285 17499 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 11:43:19.721488 17494 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 11:43:19.721812 17494 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 11:43:19.722971 17494 hybrid_clock.cc:648] HybridClock initialized: now 1781955799722958 us; error 25 us; skew 500 ppm
Jun 20 11:43:19 dist-test-slave-cjc6 krb5kdc[17253](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781955799, etypes {rep=17 tkt=17 ses=17}, kudu/127.6.234.194@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 11:43:19.729933 17494 init.cc:377] Logged in from keytab as kudu/127.6.234.194@KRBTEST.COM (short username kudu)
I20260620 11:43:19.730337 17494 webserver.cc:460] Webserver started at http://127.6.234.194:34007/ using document root <none> and password file <none>
I20260620 11:43:19.730541 17494 fs_manager.cc:359] Metadata directory not provided
I20260620 11:43:19.730581 17494 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 11:43:19.730665 17494 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 11:43:19.731662 17494 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task1JRg7N/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781955746769051-7083-0/minicluster-data/ts-1/data/instance:
uuid: "c5b9dd725bb24fdca8d66cf909706f38"
format_stamp: "Formatted at 2026-06-20 11:43:19 on dist-test-slave-cjc6"
server_key: "ff502084724f527b0866b97303865ec6"
server_key_iv: "dbd29de948a9dcd14a971f1cd9ee9505"
server_key_version: "encryptionkey@0"
I20260620 11:43:19.731953 17494 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task1JRg7N/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781955746769051-7083-0/minicluster-data/ts-1/wal/instance:
uuid: "c5b9dd725bb24fdca8d66cf909706f38"
format_stamp: "Formatted at 2026-06-20 11:43:19 on dist-test-slave-cjc6"
server_key: "ff502084724f527b0866b97303865ec6"
server_key_iv: "dbd29de948a9dcd14a971f1cd9ee9505"
server_key_version: "encryptionkey@0"
I20260620 11:43:19.733247 17494 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260620 11:43:19.734086 17509 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 11:43:19.734265 17494 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.000s
I20260620 11:43:19.734339 17494 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task1JRg7N/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781955746769051-7083-0/minicluster-data/ts-1/data,/tmp/dist-test-task1JRg7N/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781955746769051-7083-0/minicluster-data/ts-1/wal
uuid: "c5b9dd725bb24fdca8d66cf909706f38"
format_stamp: "Formatted at 2026-06-20 11:43:19 on dist-test-slave-cjc6"
server_key: "ff502084724f527b0866b97303865ec6"
server_key_iv: "dbd29de948a9dcd14a971f1cd9ee9505"
server_key_version: "encryptionkey@0"
I20260620 11:43:19.734391 17494 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task1JRg7N/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781955746769051-7083-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-task1JRg7N/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781955746769051-7083-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-task1JRg7N/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781955746769051-7083-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 11:43:19.744197 17494 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 11:43:19.745952 17494 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 11:43:19.746055 17494 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 11:43:19.746217 17494 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 11:43:19.746454 17494 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 11:43:19.746483 17494 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 11:43:19.746502 17494 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 11:43:19.746515 17494 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 11:43:19.752030 17494 rpc_server.cc:307] RPC server started. Bound to: 127.6.234.194:41679
I20260620 11:43:19.752104 17622 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.6.234.194:41679 every 8 connection(s)
I20260620 11:43:19.752415 17494 server_base.cc:1166] Dumped server information to /tmp/dist-test-task1JRg7N/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781955746769051-7083-0/minicluster-data/ts-1/data/info.pb
Jun 20 11:43:19 dist-test-slave-cjc6 krb5kdc[17253](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781955799, etypes {rep=17 tkt=17 ses=17}, kudu/127.6.234.194@KRBTEST.COM for kudu/127.6.234.254@KRBTEST.COM
I20260620 11:43:19.758605 7083 external_mini_cluster.cc:1388] Started /tmp/dist-test-task1JRg7N/build/release/bin/kudu as pid 17494
I20260620 11:43:19.758715 7083 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task1JRg7N/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781955746769051-7083-0/minicluster-data/ts-1/wal/instance
I20260620 11:43:19.758916 7083 external_mini_cluster.cc:1428] Setting key d57a0aae58657851224c935929ac74ec
I20260620 11:43:19.761896 17338 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 11:43:19.753150 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.6.234.194:38151 (local address 127.6.234.254:40311)
0620 11:43:19.753201 (+ 51us) server_negotiation.cc:206] Beginning negotiation
0620 11:43:19.753204 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0620 11:43:19.753412 (+ 208us) server_negotiation.cc:407] Connection header received
0620 11:43:19.754019 (+ 607us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 11:43:19.754020 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 11:43:19.754031 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 11:43:19.754093 (+ 62us) server_negotiation.cc:226] Negotiated authn=SASL
0620 11:43:19.754986 (+ 893us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 11:43:19.755475 (+ 489us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 11:43:19.756443 (+ 968us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 11:43:19.756642 (+ 199us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 11:43:19.756703 (+ 61us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 11:43:19.758974 (+ 2271us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 11:43:19.758976 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 11:43:19.758977 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 11:43:19.758984 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0620 11:43:19.760393 (+ 1409us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 11:43:19.760897 (+ 504us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 11:43:19.760898 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 11:43:19.760899 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 11:43:19.760951 (+ 52us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 11:43:19.761230 (+ 279us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 11:43:19.761231 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 11:43:19.761231 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 11:43:19.761345 (+ 114us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 11:43:19.761415 (+ 70us) server_negotiation.cc:1035] Waiting for connection context
0620 11:43:19.761717 (+ 302us) server_negotiation.cc:299] Negotiation successful
0620 11:43:19.761804 (+ 87us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":19}
I20260620 11:43:19.762095 17625 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 11:43:19.753137 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.234.254:40311 (local address 127.6.234.194:38151)
0620 11:43:19.753294 (+ 157us) negotiation.cc:107] Waiting for socket to connect
0620 11:43:19.753327 (+ 33us) client_negotiation.cc:174] Beginning negotiation
0620 11:43:19.753919 (+ 592us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 11:43:19.754161 (+ 242us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 11:43:19.754164 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 11:43:19.754721 (+ 557us) client_negotiation.cc:189] Negotiated authn=SASL
0620 11:43:19.754913 (+ 192us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 11:43:19.754917 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 11:43:19.755569 (+ 652us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 11:43:19.755570 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 11:43:19.756360 (+ 790us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 11:43:19.756362 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 11:43:19.756737 (+ 375us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 11:43:19.756738 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 11:43:19.756843 (+ 105us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 11:43:19.757376 (+ 533us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 11:43:19.757384 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0620 11:43:19.758867 (+ 1483us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 11:43:19.760474 (+ 1607us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 11:43:19.760480 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 11:43:19.760482 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0620 11:43:19.760822 (+ 340us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 11:43:19.761026 (+ 204us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 11:43:19.761027 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 11:43:19.761028 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0620 11:43:19.761148 (+ 120us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 11:43:19.761418 (+ 270us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 11:43:19.761425 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 11:43:19.761561 (+ 136us) client_negotiation.cc:769] Sending connection context
0620 11:43:19.761674 (+ 113us) client_negotiation.cc:240] Negotiation successful
0620 11:43:19.761860 (+ 186us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":121,"thread_start_us":83,"threads_started":1}
I20260620 11:43:19.762552 17623 heartbeater.cc:347] Connected to a master server at 127.6.234.254:40311
I20260620 11:43:19.762647 17623 heartbeater.cc:464] Registering TS with master...
I20260620 11:43:19.762835 17623 heartbeater.cc:511] Master 127.6.234.254:40311 requested a full tablet report, sending...
I20260620 11:43:19.763219 17298 ts_manager.cc:194] Registered new tserver with Master: c5b9dd725bb24fdca8d66cf909706f38 (127.6.234.194:41679)
I20260620 11:43:19.763764 17298 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.6.234.194@KRBTEST.COM'} at 127.6.234.194:38151
WARNING: no policy specified for kudu/127.6.234.195@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.6.234.195@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.6.234.195 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task1JRg7N/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781955746769051-7083-0/krb5kdc/kudu_127.6.234.195.keytab.
Entry for principal kudu/127.6.234.195 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task1JRg7N/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781955746769051-7083-0/krb5kdc/kudu_127.6.234.195.keytab.
I20260620 11:43:19.782661 7083 external_mini_cluster.cc:1326] Running /tmp/dist-test-task1JRg7N/build/release/bin/kudu
/tmp/dist-test-task1JRg7N/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task1JRg7N/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781955746769051-7083-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-task1JRg7N/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781955746769051-7083-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task1JRg7N/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781955746769051-7083-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-task1JRg7N/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781955746769051-7083-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.6.234.195:0
--local_ip_for_outbound_sockets=127.6.234.195
--webserver_interface=127.6.234.195
--webserver_port=0
--tserver_master_addrs=127.6.234.254:40311
--keytab_file=/tmp/dist-test-task1JRg7N/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781955746769051-7083-0/krb5kdc/kudu_127.6.234.195.keytab
--principal=kudu/127.6.234.195
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.6.234.212:43989
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task1JRg7N/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781955746769051-7083-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task1JRg7N/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781955746769051-7083-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task1JRg7N/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781955746769051-7083-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 11:43:19.862593 17628 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 11:43:19.862784 17628 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 11:43:19.862807 17628 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 11:43:19.863874 17628 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 11:43:19.863907 17628 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 11:43:19.863932 17628 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.6.234.195
I20260620 11:43:19.865370 17628 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.6.234.212:43989
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task1JRg7N/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781955746769051-7083-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-task1JRg7N/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781955746769051-7083-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task1JRg7N/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781955746769051-7083-0/krb5kdc/kudu_127.6.234.195.keytab
--principal=kudu/127.6.234.195
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.6.234.195:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task1JRg7N/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781955746769051-7083-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.6.234.195
--webserver_port=0
--tserver_master_addrs=127.6.234.254:40311
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.17628
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.6.234.195
--log_dir=/tmp/dist-test-task1JRg7N/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781955746769051-7083-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 20 Jun 2026 10:32:37 UTC on ccycloud.quasar-bgyofv.root.comops.site
build id 4457173
I20260620 11:43:19.865581 17628 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 11:43:19.865818 17628 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 11:43:19.870841 17633 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 11:43:19.870901 17628 server_base.cc:1034] running on GCE node
W20260620 11:43:19.871061 17636 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 11:43:19.871060 17634 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 11:43:19.871462 17628 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 11:43:19.871764 17628 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 11:43:19.872884 17628 hybrid_clock.cc:648] HybridClock initialized: now 1781955799872858 us; error 37 us; skew 500 ppm
Jun 20 11:43:19 dist-test-slave-cjc6 krb5kdc[17253](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781955799, etypes {rep=17 tkt=17 ses=17}, kudu/127.6.234.195@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 11:43:19.875744 17628 init.cc:377] Logged in from keytab as kudu/127.6.234.195@KRBTEST.COM (short username kudu)
I20260620 11:43:19.876183 17628 webserver.cc:460] Webserver started at http://127.6.234.195:38059/ using document root <none> and password file <none>
I20260620 11:43:19.876428 17628 fs_manager.cc:359] Metadata directory not provided
I20260620 11:43:19.876497 17628 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 11:43:19.876652 17628 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 11:43:19.877646 17628 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task1JRg7N/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781955746769051-7083-0/minicluster-data/ts-2/data/instance:
uuid: "682c59e72daf43a8a4192fb904a464cc"
format_stamp: "Formatted at 2026-06-20 11:43:19 on dist-test-slave-cjc6"
server_key: "a769dfa7f982af09f9ecc024d353e113"
server_key_iv: "b91bcf58b2330f50904997a72a5c1280"
server_key_version: "encryptionkey@0"
I20260620 11:43:19.878072 17628 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task1JRg7N/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781955746769051-7083-0/minicluster-data/ts-2/wal/instance:
uuid: "682c59e72daf43a8a4192fb904a464cc"
format_stamp: "Formatted at 2026-06-20 11:43:19 on dist-test-slave-cjc6"
server_key: "a769dfa7f982af09f9ecc024d353e113"
server_key_iv: "b91bcf58b2330f50904997a72a5c1280"
server_key_version: "encryptionkey@0"
I20260620 11:43:19.879352 17628 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.001s sys 0.000s
I20260620 11:43:19.880059 17643 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 11:43:19.880227 17628 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260620 11:43:19.880326 17628 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task1JRg7N/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781955746769051-7083-0/minicluster-data/ts-2/data,/tmp/dist-test-task1JRg7N/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781955746769051-7083-0/minicluster-data/ts-2/wal
uuid: "682c59e72daf43a8a4192fb904a464cc"
format_stamp: "Formatted at 2026-06-20 11:43:19 on dist-test-slave-cjc6"
server_key: "a769dfa7f982af09f9ecc024d353e113"
server_key_iv: "b91bcf58b2330f50904997a72a5c1280"
server_key_version: "encryptionkey@0"
I20260620 11:43:19.880412 17628 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task1JRg7N/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781955746769051-7083-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-task1JRg7N/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781955746769051-7083-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-task1JRg7N/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781955746769051-7083-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 11:43:19.902335 17628 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 11:43:19.904273 17628 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 11:43:19.904438 17628 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 11:43:19.904664 17628 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 11:43:19.904983 17628 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 11:43:19.905037 17628 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 11:43:19.905081 17628 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 11:43:19.905112 17628 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 11:43:19.911026 17628 rpc_server.cc:307] RPC server started. Bound to: 127.6.234.195:40947
I20260620 11:43:19.911073 17756 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.6.234.195:40947 every 8 connection(s)
I20260620 11:43:19.911393 17628 server_base.cc:1166] Dumped server information to /tmp/dist-test-task1JRg7N/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781955746769051-7083-0/minicluster-data/ts-2/data/info.pb
I20260620 11:43:19.917484 7083 external_mini_cluster.cc:1388] Started /tmp/dist-test-task1JRg7N/build/release/bin/kudu as pid 17628
I20260620 11:43:19.917586 7083 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task1JRg7N/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781955746769051-7083-0/minicluster-data/ts-2/wal/instance
I20260620 11:43:19.917814 7083 external_mini_cluster.cc:1428] Setting key 8d43f58dd3a88523d3c6ea0ef979cb39
Jun 20 11:43:19 dist-test-slave-cjc6 krb5kdc[17253](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781955799, etypes {rep=17 tkt=17 ses=17}, kudu/127.6.234.195@KRBTEST.COM for kudu/127.6.234.254@KRBTEST.COM
I20260620 11:43:19.921034 17338 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 11:43:19.912112 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.6.234.195:54553 (local address 127.6.234.254:40311)
0620 11:43:19.912163 (+ 51us) server_negotiation.cc:206] Beginning negotiation
0620 11:43:19.912164 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0620 11:43:19.912368 (+ 204us) server_negotiation.cc:407] Connection header received
0620 11:43:19.912898 (+ 530us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 11:43:19.912899 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 11:43:19.912910 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 11:43:19.912975 (+ 65us) server_negotiation.cc:226] Negotiated authn=SASL
0620 11:43:19.914002 (+ 1027us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 11:43:19.914544 (+ 542us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 11:43:19.915620 (+ 1076us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 11:43:19.915817 (+ 197us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 11:43:19.915931 (+ 114us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 11:43:19.918403 (+ 2472us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 11:43:19.918405 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 11:43:19.918405 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 11:43:19.918412 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0620 11:43:19.919923 (+ 1511us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 11:43:19.920292 (+ 369us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 11:43:19.920293 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 11:43:19.920293 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 11:43:19.920343 (+ 50us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 11:43:19.920548 (+ 205us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 11:43:19.920549 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 11:43:19.920549 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 11:43:19.920659 (+ 110us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 11:43:19.920709 (+ 50us) server_negotiation.cc:1035] Waiting for connection context
0620 11:43:19.920919 (+ 210us) server_negotiation.cc:299] Negotiation successful
0620 11:43:19.920974 (+ 55us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":24}
I20260620 11:43:19.921250 17759 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 11:43:19.912064 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.234.254:40311 (local address 127.6.234.195:54553)
0620 11:43:19.912253 (+ 189us) negotiation.cc:107] Waiting for socket to connect
0620 11:43:19.912295 (+ 42us) client_negotiation.cc:174] Beginning negotiation
0620 11:43:19.912782 (+ 487us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 11:43:19.913056 (+ 274us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 11:43:19.913059 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 11:43:19.913623 (+ 564us) client_negotiation.cc:189] Negotiated authn=SASL
0620 11:43:19.913918 (+ 295us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 11:43:19.913923 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 11:43:19.914687 (+ 764us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 11:43:19.914689 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 11:43:19.915509 (+ 820us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 11:43:19.915511 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 11:43:19.915940 (+ 429us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 11:43:19.915941 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 11:43:19.916088 (+ 147us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 11:43:19.916607 (+ 519us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 11:43:19.916615 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0620 11:43:19.918311 (+ 1696us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 11:43:19.919994 (+ 1683us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 11:43:19.919999 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 11:43:19.920001 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0620 11:43:19.920225 (+ 224us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 11:43:19.920402 (+ 177us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 11:43:19.920403 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 11:43:19.920403 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0620 11:43:19.920496 (+ 93us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 11:43:19.920715 (+ 219us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 11:43:19.920719 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 11:43:19.920829 (+ 110us) client_negotiation.cc:769] Sending connection context
0620 11:43:19.920910 (+ 81us) client_negotiation.cc:240] Negotiation successful
0620 11:43:19.921025 (+ 115us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":146,"thread_start_us":103,"threads_started":1}
I20260620 11:43:19.921772 17757 heartbeater.cc:347] Connected to a master server at 127.6.234.254:40311
I20260620 11:43:19.921864 17757 heartbeater.cc:464] Registering TS with master...
I20260620 11:43:19.921995 17757 heartbeater.cc:511] Master 127.6.234.254:40311 requested a full tablet report, sending...
I20260620 11:43:19.922346 17298 ts_manager.cc:194] Registered new tserver with Master: 682c59e72daf43a8a4192fb904a464cc (127.6.234.195:40947)
I20260620 11:43:19.922724 17298 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.6.234.195@KRBTEST.COM'} at 127.6.234.195:54553
I20260620 11:43:19.929662 7083 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260620 11:43:19.936563 17338 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 11:43:19.930947 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48994 (local address 127.6.234.254:40311)
0620 11:43:19.931013 (+ 66us) server_negotiation.cc:206] Beginning negotiation
0620 11:43:19.931015 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 11:43:19.931142 (+ 127us) server_negotiation.cc:407] Connection header received
0620 11:43:19.931177 (+ 35us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 11:43:19.931178 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 11:43:19.931187 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 11:43:19.931232 (+ 45us) server_negotiation.cc:226] Negotiated authn=SASL
0620 11:43:19.931757 (+ 525us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 11:43:19.932214 (+ 457us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 11:43:19.932884 (+ 670us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 11:43:19.933051 (+ 167us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 11:43:19.933106 (+ 55us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 11:43:19.934247 (+ 1141us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 11:43:19.934248 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 11:43:19.934249 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 11:43:19.934254 (+ 5us) server_negotiation.cc:922] calling sasl_server_start()
0620 11:43:19.935508 (+ 1254us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 11:43:19.935821 (+ 313us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 11:43:19.935821 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 11:43:19.935822 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 11:43:19.935867 (+ 45us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 11:43:19.936024 (+ 157us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 11:43:19.936024 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 11:43:19.936024 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 11:43:19.936157 (+ 133us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 11:43:19.936227 (+ 70us) server_negotiation.cc:1035] Waiting for connection context
0620 11:43:19.936396 (+ 169us) server_negotiation.cc:299] Negotiation successful
0620 11:43:19.936464 (+ 68us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":38}
I20260620 11:43:19.938262 17298 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:48994:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260620 11:43:19.939788 17298 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260620 11:43:19.946471 17772 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 11:43:19.942290 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42660 (local address 127.6.234.194:41679)
0620 11:43:19.942968 (+ 678us) server_negotiation.cc:206] Beginning negotiation
0620 11:43:19.942973 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 11:43:19.942982 (+ 9us) server_negotiation.cc:407] Connection header received
0620 11:43:19.943018 (+ 36us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 11:43:19.943020 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 11:43:19.943034 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 11:43:19.943110 (+ 76us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 11:43:19.943228 (+ 118us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 11:43:19.943908 (+ 680us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 11:43:19.945492 (+ 1584us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 11:43:19.946066 (+ 574us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 11:43:19.946158 (+ 92us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 11:43:19.946215 (+ 57us) server_negotiation.cc:1035] Waiting for connection context
0620 11:43:19.946344 (+ 129us) server_negotiation.cc:299] Negotiation successful
0620 11:43:19.946382 (+ 38us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":644,"thread_start_us":67,"threads_started":1}
I20260620 11:43:19.946511 17769 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 11:43:19.942079 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.234.194:41679 (local address 127.0.0.1:42660)
0620 11:43:19.942741 (+ 662us) negotiation.cc:107] Waiting for socket to connect
0620 11:43:19.942815 (+ 74us) client_negotiation.cc:174] Beginning negotiation
0620 11:43:19.942855 (+ 40us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 11:43:19.943124 (+ 269us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 11:43:19.943125 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 11:43:19.943127 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 11:43:19.943168 (+ 41us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 11:43:19.943170 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 11:43:19.944008 (+ 838us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 11:43:19.944009 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 11:43:19.945389 (+ 1380us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 11:43:19.945391 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 11:43:19.946162 (+ 771us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 11:43:19.946163 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 11:43:19.946240 (+ 77us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 11:43:19.946253 (+ 13us) client_negotiation.cc:769] Sending connection context
0620 11:43:19.946326 (+ 73us) client_negotiation.cc:240] Negotiation successful
0620 11:43:19.946381 (+ 55us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":647,"mutex_wait_us":82,"thread_start_us":91,"threads_started":1}
I20260620 11:43:19.946620 17768 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 11:43:19.942065 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.234.193:42267 (local address 127.0.0.1:33352)
0620 11:43:19.942303 (+ 238us) negotiation.cc:107] Waiting for socket to connect
0620 11:43:19.942363 (+ 60us) client_negotiation.cc:174] Beginning negotiation
0620 11:43:19.942429 (+ 66us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 11:43:19.942640 (+ 211us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 11:43:19.942643 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 11:43:19.942645 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 11:43:19.942722 (+ 77us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 11:43:19.942724 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 11:43:19.944094 (+ 1370us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 11:43:19.944095 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 11:43:19.945390 (+ 1295us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 11:43:19.945391 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 11:43:19.946311 (+ 920us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 11:43:19.946311 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 11:43:19.946372 (+ 61us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 11:43:19.946376 (+ 4us) client_negotiation.cc:769] Sending connection context
0620 11:43:19.946419 (+ 43us) client_negotiation.cc:240] Negotiation successful
0620 11:43:19.946454 (+ 35us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":219,"mutex_wait_us":84,"thread_start_us":55,"threads_started":1}
I20260620 11:43:19.946697 17770 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 11:43:19.942068 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.234.195:40947 (local address 127.0.0.1:40734)
0620 11:43:19.942577 (+ 509us) negotiation.cc:107] Waiting for socket to connect
0620 11:43:19.942587 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0620 11:43:19.942654 (+ 67us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 11:43:19.942878 (+ 224us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 11:43:19.942879 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 11:43:19.942881 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 11:43:19.942925 (+ 44us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 11:43:19.942926 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 11:43:19.944219 (+ 1293us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 11:43:19.944220 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 11:43:19.945232 (+ 1012us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 11:43:19.945235 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 11:43:19.946253 (+ 1018us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 11:43:19.946254 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 11:43:19.946319 (+ 65us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 11:43:19.946323 (+ 4us) client_negotiation.cc:769] Sending connection context
0620 11:43:19.946371 (+ 48us) client_negotiation.cc:240] Negotiation successful
0620 11:43:19.946419 (+ 48us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":489,"thread_start_us":97,"threads_started":1}
I20260620 11:43:19.946713 17773 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 11:43:19.942405 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40734 (local address 127.6.234.195:40947)
0620 11:43:19.942562 (+ 157us) server_negotiation.cc:206] Beginning negotiation
0620 11:43:19.942567 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 11:43:19.942641 (+ 74us) server_negotiation.cc:407] Connection header received
0620 11:43:19.942729 (+ 88us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 11:43:19.942732 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 11:43:19.942747 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 11:43:19.942832 (+ 85us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 11:43:19.943044 (+ 212us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 11:43:19.944110 (+ 1066us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 11:43:19.945316 (+ 1206us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 11:43:19.946081 (+ 765us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 11:43:19.946154 (+ 73us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 11:43:19.946215 (+ 61us) server_negotiation.cc:1035] Waiting for connection context
0620 11:43:19.946555 (+ 340us) server_negotiation.cc:299] Negotiation successful
0620 11:43:19.946608 (+ 53us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":123,"thread_start_us":88,"threads_started":1}
I20260620 11:43:19.947429 17691 tablet_service.cc:1467] Processing CreateTablet for tablet 8d70c36056d24c3fa6875034ba7d04aa (DEFAULT_TABLE table=test-table [id=98edd42b61ab4487a936fb96cf1d45ec]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 11:43:19.947535 17557 tablet_service.cc:1467] Processing CreateTablet for tablet 8d70c36056d24c3fa6875034ba7d04aa (DEFAULT_TABLE table=test-table [id=98edd42b61ab4487a936fb96cf1d45ec]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 11:43:19.947804 17557 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 8d70c36056d24c3fa6875034ba7d04aa. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 11:43:19.947804 17691 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 8d70c36056d24c3fa6875034ba7d04aa. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 11:43:19.951094 17775 tablet_bootstrap.cc:492] T 8d70c36056d24c3fa6875034ba7d04aa P c5b9dd725bb24fdca8d66cf909706f38: Bootstrap starting.
I20260620 11:43:19.951246 17771 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 11:43:19.942228 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33352 (local address 127.6.234.193:42267)
0620 11:43:19.942415 (+ 187us) server_negotiation.cc:206] Beginning negotiation
0620 11:43:19.942420 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 11:43:19.942431 (+ 11us) server_negotiation.cc:407] Connection header received
0620 11:43:19.942515 (+ 84us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 11:43:19.942518 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 11:43:19.942533 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 11:43:19.942622 (+ 89us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 11:43:19.942807 (+ 185us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 11:43:19.944007 (+ 1200us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 11:43:19.945476 (+ 1469us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 11:43:19.946179 (+ 703us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 11:43:19.946238 (+ 59us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 11:43:19.946283 (+ 45us) server_negotiation.cc:1035] Waiting for connection context
0620 11:43:19.947166 (+ 883us) server_negotiation.cc:299] Negotiation successful
0620 11:43:19.951165 (+ 3999us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":146,"thread_start_us":82,"threads_started":1}
I20260620 11:43:19.951879 17775 tablet_bootstrap.cc:654] T 8d70c36056d24c3fa6875034ba7d04aa P c5b9dd725bb24fdca8d66cf909706f38: Neither blocks nor log segments found. Creating new log.
I20260620 11:43:19.951881 17423 tablet_service.cc:1467] Processing CreateTablet for tablet 8d70c36056d24c3fa6875034ba7d04aa (DEFAULT_TABLE table=test-table [id=98edd42b61ab4487a936fb96cf1d45ec]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 11:43:19.952252 17423 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 8d70c36056d24c3fa6875034ba7d04aa. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 11:43:19.952442 17775 log.cc:824] T 8d70c36056d24c3fa6875034ba7d04aa P c5b9dd725bb24fdca8d66cf909706f38: Log is configured to *not* fsync() on all Append() calls
I20260620 11:43:19.952785 17774 tablet_bootstrap.cc:492] T 8d70c36056d24c3fa6875034ba7d04aa P 682c59e72daf43a8a4192fb904a464cc: Bootstrap starting.
I20260620 11:43:19.953212 17775 tablet_bootstrap.cc:492] T 8d70c36056d24c3fa6875034ba7d04aa P c5b9dd725bb24fdca8d66cf909706f38: No bootstrap required, opened a new log
I20260620 11:43:19.953281 17775 ts_tablet_manager.cc:1397] T 8d70c36056d24c3fa6875034ba7d04aa P c5b9dd725bb24fdca8d66cf909706f38: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20260620 11:43:19.953413 17774 tablet_bootstrap.cc:654] T 8d70c36056d24c3fa6875034ba7d04aa P 682c59e72daf43a8a4192fb904a464cc: Neither blocks nor log segments found. Creating new log.
I20260620 11:43:19.954020 17774 log.cc:824] T 8d70c36056d24c3fa6875034ba7d04aa P 682c59e72daf43a8a4192fb904a464cc: Log is configured to *not* fsync() on all Append() calls
I20260620 11:43:19.954802 17774 tablet_bootstrap.cc:492] T 8d70c36056d24c3fa6875034ba7d04aa P 682c59e72daf43a8a4192fb904a464cc: No bootstrap required, opened a new log
I20260620 11:43:19.954870 17774 ts_tablet_manager.cc:1397] T 8d70c36056d24c3fa6875034ba7d04aa P 682c59e72daf43a8a4192fb904a464cc: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20260620 11:43:19.955325 17778 tablet_bootstrap.cc:492] T 8d70c36056d24c3fa6875034ba7d04aa P b36a5c84137e48baaa4d49f2b9dcf6e8: Bootstrap starting.
I20260620 11:43:19.955351 17775 raft_consensus.cc:348] T 8d70c36056d24c3fa6875034ba7d04aa P c5b9dd725bb24fdca8d66cf909706f38 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b36a5c84137e48baaa4d49f2b9dcf6e8" member_type: VOTER last_known_addr { host: "127.6.234.193" port: 42267 } } peers { permanent_uuid: "682c59e72daf43a8a4192fb904a464cc" member_type: VOTER last_known_addr { host: "127.6.234.195" port: 40947 } } peers { permanent_uuid: "c5b9dd725bb24fdca8d66cf909706f38" member_type: VOTER last_known_addr { host: "127.6.234.194" port: 41679 } }
I20260620 11:43:19.955533 17775 raft_consensus.cc:374] T 8d70c36056d24c3fa6875034ba7d04aa P c5b9dd725bb24fdca8d66cf909706f38 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 11:43:19.955574 17775 raft_consensus.cc:729] T 8d70c36056d24c3fa6875034ba7d04aa P c5b9dd725bb24fdca8d66cf909706f38 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c5b9dd725bb24fdca8d66cf909706f38, State: Initialized, Role: FOLLOWER
I20260620 11:43:19.955696 17775 consensus_queue.cc:260] T 8d70c36056d24c3fa6875034ba7d04aa P c5b9dd725bb24fdca8d66cf909706f38 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b36a5c84137e48baaa4d49f2b9dcf6e8" member_type: VOTER last_known_addr { host: "127.6.234.193" port: 42267 } } peers { permanent_uuid: "682c59e72daf43a8a4192fb904a464cc" member_type: VOTER last_known_addr { host: "127.6.234.195" port: 40947 } } peers { permanent_uuid: "c5b9dd725bb24fdca8d66cf909706f38" member_type: VOTER last_known_addr { host: "127.6.234.194" port: 41679 } }
I20260620 11:43:19.955957 17775 ts_tablet_manager.cc:1428] T 8d70c36056d24c3fa6875034ba7d04aa P c5b9dd725bb24fdca8d66cf909706f38: Time spent starting tablet: real 0.003s user 0.003s sys 0.000s
I20260620 11:43:19.956103 17623 heartbeater.cc:503] Master 127.6.234.254:40311 was elected leader, sending a full tablet report...
I20260620 11:43:19.956178 17778 tablet_bootstrap.cc:654] T 8d70c36056d24c3fa6875034ba7d04aa P b36a5c84137e48baaa4d49f2b9dcf6e8: Neither blocks nor log segments found. Creating new log.
I20260620 11:43:19.956730 17778 log.cc:824] T 8d70c36056d24c3fa6875034ba7d04aa P b36a5c84137e48baaa4d49f2b9dcf6e8: Log is configured to *not* fsync() on all Append() calls
I20260620 11:43:19.956717 17774 raft_consensus.cc:348] T 8d70c36056d24c3fa6875034ba7d04aa P 682c59e72daf43a8a4192fb904a464cc [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b36a5c84137e48baaa4d49f2b9dcf6e8" member_type: VOTER last_known_addr { host: "127.6.234.193" port: 42267 } } peers { permanent_uuid: "682c59e72daf43a8a4192fb904a464cc" member_type: VOTER last_known_addr { host: "127.6.234.195" port: 40947 } } peers { permanent_uuid: "c5b9dd725bb24fdca8d66cf909706f38" member_type: VOTER last_known_addr { host: "127.6.234.194" port: 41679 } }
I20260620 11:43:19.956871 17774 raft_consensus.cc:374] T 8d70c36056d24c3fa6875034ba7d04aa P 682c59e72daf43a8a4192fb904a464cc [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 11:43:19.956915 17774 raft_consensus.cc:729] T 8d70c36056d24c3fa6875034ba7d04aa P 682c59e72daf43a8a4192fb904a464cc [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 682c59e72daf43a8a4192fb904a464cc, State: Initialized, Role: FOLLOWER
I20260620 11:43:19.957029 17774 consensus_queue.cc:260] T 8d70c36056d24c3fa6875034ba7d04aa P 682c59e72daf43a8a4192fb904a464cc [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b36a5c84137e48baaa4d49f2b9dcf6e8" member_type: VOTER last_known_addr { host: "127.6.234.193" port: 42267 } } peers { permanent_uuid: "682c59e72daf43a8a4192fb904a464cc" member_type: VOTER last_known_addr { host: "127.6.234.195" port: 40947 } } peers { permanent_uuid: "c5b9dd725bb24fdca8d66cf909706f38" member_type: VOTER last_known_addr { host: "127.6.234.194" port: 41679 } }
I20260620 11:43:19.957314 17774 ts_tablet_manager.cc:1428] T 8d70c36056d24c3fa6875034ba7d04aa P 682c59e72daf43a8a4192fb904a464cc: Time spent starting tablet: real 0.002s user 0.003s sys 0.000s
I20260620 11:43:19.957463 17757 heartbeater.cc:503] Master 127.6.234.254:40311 was elected leader, sending a full tablet report...
I20260620 11:43:19.957646 17778 tablet_bootstrap.cc:492] T 8d70c36056d24c3fa6875034ba7d04aa P b36a5c84137e48baaa4d49f2b9dcf6e8: No bootstrap required, opened a new log
I20260620 11:43:19.957743 17778 ts_tablet_manager.cc:1397] T 8d70c36056d24c3fa6875034ba7d04aa P b36a5c84137e48baaa4d49f2b9dcf6e8: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260620 11:43:19.959311 17778 raft_consensus.cc:348] T 8d70c36056d24c3fa6875034ba7d04aa P b36a5c84137e48baaa4d49f2b9dcf6e8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b36a5c84137e48baaa4d49f2b9dcf6e8" member_type: VOTER last_known_addr { host: "127.6.234.193" port: 42267 } } peers { permanent_uuid: "682c59e72daf43a8a4192fb904a464cc" member_type: VOTER last_known_addr { host: "127.6.234.195" port: 40947 } } peers { permanent_uuid: "c5b9dd725bb24fdca8d66cf909706f38" member_type: VOTER last_known_addr { host: "127.6.234.194" port: 41679 } }
I20260620 11:43:19.959434 17778 raft_consensus.cc:374] T 8d70c36056d24c3fa6875034ba7d04aa P b36a5c84137e48baaa4d49f2b9dcf6e8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 11:43:19.959456 17778 raft_consensus.cc:729] T 8d70c36056d24c3fa6875034ba7d04aa P b36a5c84137e48baaa4d49f2b9dcf6e8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b36a5c84137e48baaa4d49f2b9dcf6e8, State: Initialized, Role: FOLLOWER
I20260620 11:43:19.959560 17778 consensus_queue.cc:260] T 8d70c36056d24c3fa6875034ba7d04aa P b36a5c84137e48baaa4d49f2b9dcf6e8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b36a5c84137e48baaa4d49f2b9dcf6e8" member_type: VOTER last_known_addr { host: "127.6.234.193" port: 42267 } } peers { permanent_uuid: "682c59e72daf43a8a4192fb904a464cc" member_type: VOTER last_known_addr { host: "127.6.234.195" port: 40947 } } peers { permanent_uuid: "c5b9dd725bb24fdca8d66cf909706f38" member_type: VOTER last_known_addr { host: "127.6.234.194" port: 41679 } }
I20260620 11:43:19.959786 17778 ts_tablet_manager.cc:1428] T 8d70c36056d24c3fa6875034ba7d04aa P b36a5c84137e48baaa4d49f2b9dcf6e8: Time spent starting tablet: real 0.002s user 0.000s sys 0.002s
I20260620 11:43:19.959859 17489 heartbeater.cc:503] Master 127.6.234.254:40311 was elected leader, sending a full tablet report...
I20260620 11:43:19.964344 17781 raft_consensus.cc:482] T 8d70c36056d24c3fa6875034ba7d04aa P 682c59e72daf43a8a4192fb904a464cc [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260620 11:43:19.964424 17781 raft_consensus.cc:504] T 8d70c36056d24c3fa6875034ba7d04aa P 682c59e72daf43a8a4192fb904a464cc [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b36a5c84137e48baaa4d49f2b9dcf6e8" member_type: VOTER last_known_addr { host: "127.6.234.193" port: 42267 } } peers { permanent_uuid: "682c59e72daf43a8a4192fb904a464cc" member_type: VOTER last_known_addr { host: "127.6.234.195" port: 40947 } } peers { permanent_uuid: "c5b9dd725bb24fdca8d66cf909706f38" member_type: VOTER last_known_addr { host: "127.6.234.194" port: 41679 } }
I20260620 11:43:19.964684 17781 leader_election.cc:288] T 8d70c36056d24c3fa6875034ba7d04aa P 682c59e72daf43a8a4192fb904a464cc [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers b36a5c84137e48baaa4d49f2b9dcf6e8 (127.6.234.193:42267), c5b9dd725bb24fdca8d66cf909706f38 (127.6.234.194:41679)
I20260620 11:43:19.967742 17771 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 11:43:19.964964 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.6.234.195:40829 (local address 127.6.234.193:42267)
0620 11:43:19.965023 (+ 59us) server_negotiation.cc:206] Beginning negotiation
0620 11:43:19.965025 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 11:43:19.965059 (+ 34us) server_negotiation.cc:407] Connection header received
0620 11:43:19.965142 (+ 83us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 11:43:19.965143 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 11:43:19.965152 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 11:43:19.965213 (+ 61us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 11:43:19.965431 (+ 218us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 11:43:19.966020 (+ 589us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 11:43:19.967057 (+ 1037us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 11:43:19.967466 (+ 409us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 11:43:19.967533 (+ 67us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 11:43:19.967539 (+ 6us) server_negotiation.cc:1035] Waiting for connection context
0620 11:43:19.967674 (+ 135us) server_negotiation.cc:299] Negotiation successful
0620 11:43:19.967700 (+ 26us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":41}
I20260620 11:43:19.967800 17759 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 11:43:19.964837 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.234.194:41679 (local address 127.6.234.195:36691)
0620 11:43:19.964895 (+ 58us) negotiation.cc:107] Waiting for socket to connect
0620 11:43:19.964908 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0620 11:43:19.964956 (+ 48us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 11:43:19.965142 (+ 186us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 11:43:19.965143 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 11:43:19.965144 (+ 1us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 11:43:19.965213 (+ 69us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 11:43:19.965215 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 11:43:19.966003 (+ 788us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 11:43:19.966004 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 11:43:19.966847 (+ 843us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 11:43:19.966849 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 11:43:19.967516 (+ 667us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 11:43:19.967517 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 11:43:19.967597 (+ 80us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 11:43:19.967600 (+ 3us) client_negotiation.cc:769] Sending connection context
0620 11:43:19.967666 (+ 66us) client_negotiation.cc:240] Negotiation successful
0620 11:43:19.967707 (+ 41us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":36,"mutex_wait_us":2}
I20260620 11:43:19.967814 17783 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 11:43:19.964849 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.234.193:42267 (local address 127.6.234.195:40829)
0620 11:43:19.965005 (+ 156us) negotiation.cc:107] Waiting for socket to connect
0620 11:43:19.965015 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0620 11:43:19.965073 (+ 58us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 11:43:19.965220 (+ 147us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 11:43:19.965220 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 11:43:19.965222 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 11:43:19.965354 (+ 132us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 11:43:19.965356 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 11:43:19.966094 (+ 738us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 11:43:19.966095 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 11:43:19.966918 (+ 823us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 11:43:19.966938 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 11:43:19.967538 (+ 600us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 11:43:19.967538 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 11:43:19.967602 (+ 64us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 11:43:19.967605 (+ 3us) client_negotiation.cc:769] Sending connection context
0620 11:43:19.967665 (+ 60us) client_negotiation.cc:240] Negotiation successful
0620 11:43:19.967707 (+ 42us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":142,"mutex_wait_us":24,"thread_start_us":73,"threads_started":1}
I20260620 11:43:19.968006 17772 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 11:43:19.964963 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.6.234.195:36691 (local address 127.6.234.194:41679)
0620 11:43:19.965008 (+ 45us) server_negotiation.cc:206] Beginning negotiation
0620 11:43:19.965010 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 11:43:19.965021 (+ 11us) server_negotiation.cc:407] Connection header received
0620 11:43:19.965046 (+ 25us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 11:43:19.965047 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 11:43:19.965063 (+ 16us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 11:43:19.965120 (+ 57us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 11:43:19.965288 (+ 168us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 11:43:19.965937 (+ 649us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 11:43:19.966926 (+ 989us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 11:43:19.967351 (+ 425us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 11:43:19.967457 (+ 106us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 11:43:19.967465 (+ 8us) server_negotiation.cc:1035] Waiting for connection context
0620 11:43:19.967893 (+ 428us) server_negotiation.cc:299] Negotiation successful
0620 11:43:19.967944 (+ 51us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":25}
I20260620 11:43:19.968113 17443 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8d70c36056d24c3fa6875034ba7d04aa" candidate_uuid: "682c59e72daf43a8a4192fb904a464cc" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b36a5c84137e48baaa4d49f2b9dcf6e8" is_pre_election: true
I20260620 11:43:19.968245 17577 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8d70c36056d24c3fa6875034ba7d04aa" candidate_uuid: "682c59e72daf43a8a4192fb904a464cc" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c5b9dd725bb24fdca8d66cf909706f38" is_pre_election: true
I20260620 11:43:19.968284 17443 raft_consensus.cc:2454] T 8d70c36056d24c3fa6875034ba7d04aa P b36a5c84137e48baaa4d49f2b9dcf6e8 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 682c59e72daf43a8a4192fb904a464cc in term 0.
I20260620 11:43:19.968377 17577 raft_consensus.cc:2454] T 8d70c36056d24c3fa6875034ba7d04aa P c5b9dd725bb24fdca8d66cf909706f38 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 682c59e72daf43a8a4192fb904a464cc in term 0.
I20260620 11:43:19.968515 17645 leader_election.cc:302] T 8d70c36056d24c3fa6875034ba7d04aa P 682c59e72daf43a8a4192fb904a464cc [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 682c59e72daf43a8a4192fb904a464cc, b36a5c84137e48baaa4d49f2b9dcf6e8; no voters:
I20260620 11:43:19.968680 17781 raft_consensus.cc:2781] T 8d70c36056d24c3fa6875034ba7d04aa P 682c59e72daf43a8a4192fb904a464cc [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260620 11:43:19.968758 17781 raft_consensus.cc:482] T 8d70c36056d24c3fa6875034ba7d04aa P 682c59e72daf43a8a4192fb904a464cc [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260620 11:43:19.968794 17781 raft_consensus.cc:3037] T 8d70c36056d24c3fa6875034ba7d04aa P 682c59e72daf43a8a4192fb904a464cc [term 0 FOLLOWER]: Advancing to term 1
I20260620 11:43:19.969449 17781 raft_consensus.cc:504] T 8d70c36056d24c3fa6875034ba7d04aa P 682c59e72daf43a8a4192fb904a464cc [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b36a5c84137e48baaa4d49f2b9dcf6e8" member_type: VOTER last_known_addr { host: "127.6.234.193" port: 42267 } } peers { permanent_uuid: "682c59e72daf43a8a4192fb904a464cc" member_type: VOTER last_known_addr { host: "127.6.234.195" port: 40947 } } peers { permanent_uuid: "c5b9dd725bb24fdca8d66cf909706f38" member_type: VOTER last_known_addr { host: "127.6.234.194" port: 41679 } }
I20260620 11:43:19.969595 17781 leader_election.cc:288] T 8d70c36056d24c3fa6875034ba7d04aa P 682c59e72daf43a8a4192fb904a464cc [CANDIDATE]: Term 1 election: Requested vote from peers b36a5c84137e48baaa4d49f2b9dcf6e8 (127.6.234.193:42267), c5b9dd725bb24fdca8d66cf909706f38 (127.6.234.194:41679)
I20260620 11:43:19.969841 17443 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8d70c36056d24c3fa6875034ba7d04aa" candidate_uuid: "682c59e72daf43a8a4192fb904a464cc" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b36a5c84137e48baaa4d49f2b9dcf6e8"
I20260620 11:43:19.969847 17577 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8d70c36056d24c3fa6875034ba7d04aa" candidate_uuid: "682c59e72daf43a8a4192fb904a464cc" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c5b9dd725bb24fdca8d66cf909706f38"
I20260620 11:43:19.969911 17443 raft_consensus.cc:3037] T 8d70c36056d24c3fa6875034ba7d04aa P b36a5c84137e48baaa4d49f2b9dcf6e8 [term 0 FOLLOWER]: Advancing to term 1
I20260620 11:43:19.969940 17577 raft_consensus.cc:3037] T 8d70c36056d24c3fa6875034ba7d04aa P c5b9dd725bb24fdca8d66cf909706f38 [term 0 FOLLOWER]: Advancing to term 1
I20260620 11:43:19.970590 17443 raft_consensus.cc:2454] T 8d70c36056d24c3fa6875034ba7d04aa P b36a5c84137e48baaa4d49f2b9dcf6e8 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 682c59e72daf43a8a4192fb904a464cc in term 1.
I20260620 11:43:19.970592 17577 raft_consensus.cc:2454] T 8d70c36056d24c3fa6875034ba7d04aa P c5b9dd725bb24fdca8d66cf909706f38 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 682c59e72daf43a8a4192fb904a464cc in term 1.
I20260620 11:43:19.970764 17647 leader_election.cc:302] T 8d70c36056d24c3fa6875034ba7d04aa P 682c59e72daf43a8a4192fb904a464cc [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 682c59e72daf43a8a4192fb904a464cc, c5b9dd725bb24fdca8d66cf909706f38; no voters:
I20260620 11:43:19.970858 17781 raft_consensus.cc:2781] T 8d70c36056d24c3fa6875034ba7d04aa P 682c59e72daf43a8a4192fb904a464cc [term 1 FOLLOWER]: Leader election won for term 1
I20260620 11:43:19.971005 17781 raft_consensus.cc:686] T 8d70c36056d24c3fa6875034ba7d04aa P 682c59e72daf43a8a4192fb904a464cc [term 1 LEADER]: Becoming Leader. State: Replica: 682c59e72daf43a8a4192fb904a464cc, State: Running, Role: LEADER
I20260620 11:43:19.971103 17781 consensus_queue.cc:237] T 8d70c36056d24c3fa6875034ba7d04aa P 682c59e72daf43a8a4192fb904a464cc [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b36a5c84137e48baaa4d49f2b9dcf6e8" member_type: VOTER last_known_addr { host: "127.6.234.193" port: 42267 } } peers { permanent_uuid: "682c59e72daf43a8a4192fb904a464cc" member_type: VOTER last_known_addr { host: "127.6.234.195" port: 40947 } } peers { permanent_uuid: "c5b9dd725bb24fdca8d66cf909706f38" member_type: VOTER last_known_addr { host: "127.6.234.194" port: 41679 } }
I20260620 11:43:19.971827 17298 catalog_manager.cc:5495] T 8d70c36056d24c3fa6875034ba7d04aa P 682c59e72daf43a8a4192fb904a464cc reported cstate change: term changed from 0 to 1, leader changed from <none> to 682c59e72daf43a8a4192fb904a464cc (127.6.234.195). New cstate: current_term: 1 leader_uuid: "682c59e72daf43a8a4192fb904a464cc" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b36a5c84137e48baaa4d49f2b9dcf6e8" member_type: VOTER last_known_addr { host: "127.6.234.193" port: 42267 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "682c59e72daf43a8a4192fb904a464cc" member_type: VOTER last_known_addr { host: "127.6.234.195" port: 40947 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "c5b9dd725bb24fdca8d66cf909706f38" member_type: VOTER last_known_addr { host: "127.6.234.194" port: 41679 } health_report { overall_health: UNKNOWN } } }
I20260620 11:43:19.979254 17773 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 11:43:19.977036 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40740 (local address 127.6.234.195:40947)
0620 11:43:19.977086 (+ 50us) server_negotiation.cc:206] Beginning negotiation
0620 11:43:19.977088 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 11:43:19.977098 (+ 10us) server_negotiation.cc:407] Connection header received
0620 11:43:19.977159 (+ 61us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 11:43:19.977160 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 11:43:19.977174 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 11:43:19.977235 (+ 61us) server_negotiation.cc:226] Negotiated authn=TOKEN
0620 11:43:19.977341 (+ 106us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 11:43:19.977866 (+ 525us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 11:43:19.978571 (+ 705us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 11:43:19.978756 (+ 185us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 11:43:19.978820 (+ 64us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 11:43:19.978939 (+ 119us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0620 11:43:19.979027 (+ 88us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0620 11:43:19.979094 (+ 67us) server_negotiation.cc:1035] Waiting for connection context
0620 11:43:19.979112 (+ 18us) server_negotiation.cc:299] Negotiation successful
0620 11:43:19.979127 (+ 15us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":32}
I20260620 11:43:19.981030 17577 raft_consensus.cc:1261] T 8d70c36056d24c3fa6875034ba7d04aa P c5b9dd725bb24fdca8d66cf909706f38 [term 1 FOLLOWER]: Refusing update from remote peer 682c59e72daf43a8a4192fb904a464cc: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 11:43:19.981055 17443 raft_consensus.cc:1261] T 8d70c36056d24c3fa6875034ba7d04aa P b36a5c84137e48baaa4d49f2b9dcf6e8 [term 1 FOLLOWER]: Refusing update from remote peer 682c59e72daf43a8a4192fb904a464cc: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 11:43:19.981364 17784 consensus_queue.cc:1035] T 8d70c36056d24c3fa6875034ba7d04aa P 682c59e72daf43a8a4192fb904a464cc [LEADER]: Connected to new peer: Peer: permanent_uuid: "c5b9dd725bb24fdca8d66cf909706f38" member_type: VOTER last_known_addr { host: "127.6.234.194" port: 41679 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 11:43:19.981564 17781 consensus_queue.cc:1035] T 8d70c36056d24c3fa6875034ba7d04aa P 682c59e72daf43a8a4192fb904a464cc [LEADER]: Connected to new peer: Peer: permanent_uuid: "b36a5c84137e48baaa4d49f2b9dcf6e8" member_type: VOTER last_known_addr { host: "127.6.234.193" port: 42267 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 11:43:19.984576 17787 mvcc.cc:205] Tried to move back new op lower bound from 7298890956719460352 to 7298890956682342400. Current Snapshot: MvccSnapshot[applied={T|T < 7298890956719460352 or (T in {7298890956719460352})}]
I20260620 11:43:19.984819 17789 mvcc.cc:205] Tried to move back new op lower bound from 7298890956719460352 to 7298890956682342400. Current Snapshot: MvccSnapshot[applied={T|T < 7298890956719460352}]
I20260620 11:43:19.984928 17788 mvcc.cc:205] Tried to move back new op lower bound from 7298890956719460352 to 7298890956682342400. Current Snapshot: MvccSnapshot[applied={T|T < 7298890956719460352}]
I20260620 11:43:19.986238 17298 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:48994:
table { table_name: "test-table" } modify_external_catalogs: true
I20260620 11:43:19.986340 17298 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:48994:
table { table_name: "test-table" } modify_external_catalogs: true
I20260620 11:43:19.987053 17298 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 6a71c9c9868b487c85eb2b417342a378: Sending DeleteTablet for 3 replicas of tablet 8d70c36056d24c3fa6875034ba7d04aa
I20260620 11:43:19.987385 17423 tablet_service.cc:1514] Processing DeleteTablet for tablet 8d70c36056d24c3fa6875034ba7d04aa with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 11:43:19 UTC) from {username='kudu', principal='kudu/127.6.234.254@KRBTEST.COM'} at 127.0.0.1:33352
I20260620 11:43:19.987385 17691 tablet_service.cc:1514] Processing DeleteTablet for tablet 8d70c36056d24c3fa6875034ba7d04aa with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 11:43:19 UTC) from {username='kudu', principal='kudu/127.6.234.254@KRBTEST.COM'} at 127.0.0.1:40734
I20260620 11:43:19.987628 17798 tablet_replica.cc:330] T 8d70c36056d24c3fa6875034ba7d04aa P 682c59e72daf43a8a4192fb904a464cc: stopping tablet replica
I20260620 11:43:19.987634 7083 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task1JRg7N/build/release/bin/kudu with pid 17360
I20260620 11:43:19.987615 17557 tablet_service.cc:1514] Processing DeleteTablet for tablet 8d70c36056d24c3fa6875034ba7d04aa with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 11:43:19 UTC) from {username='kudu', principal='kudu/127.6.234.254@KRBTEST.COM'} at 127.0.0.1:42660
I20260620 11:43:19.987754 17798 raft_consensus.cc:2229] T 8d70c36056d24c3fa6875034ba7d04aa P 682c59e72daf43a8a4192fb904a464cc [term 1 LEADER]: Raft consensus shutting down.
I20260620 11:43:19.987884 17798 raft_consensus.cc:2258] T 8d70c36056d24c3fa6875034ba7d04aa P 682c59e72daf43a8a4192fb904a464cc [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 11:43:19.989044 17798 ts_tablet_manager.cc:1905] T 8d70c36056d24c3fa6875034ba7d04aa P 682c59e72daf43a8a4192fb904a464cc: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 11:43:19.990676 17798 ts_tablet_manager.cc:1918] T 8d70c36056d24c3fa6875034ba7d04aa P 682c59e72daf43a8a4192fb904a464cc: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260620 11:43:19.990738 17798 log.cc:1196] T 8d70c36056d24c3fa6875034ba7d04aa P 682c59e72daf43a8a4192fb904a464cc: Deleting WAL directory at /tmp/dist-test-task1JRg7N/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781955746769051-7083-0/minicluster-data/ts-2/wal/wals/8d70c36056d24c3fa6875034ba7d04aa
I20260620 11:43:19.990995 17798 ts_tablet_manager.cc:1939] T 8d70c36056d24c3fa6875034ba7d04aa P 682c59e72daf43a8a4192fb904a464cc: Deleting consensus metadata
I20260620 11:43:19.991329 17282 catalog_manager.cc:4841] TS 682c59e72daf43a8a4192fb904a464cc (127.6.234.195:40947): tablet 8d70c36056d24c3fa6875034ba7d04aa (table test-table [id=98edd42b61ab4487a936fb96cf1d45ec]) successfully deleted
W20260620 11:43:19.992828 17283 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.6.234.193:42267 (error 108)
I20260620 11:43:19.992946 17799 tablet_replica.cc:330] T 8d70c36056d24c3fa6875034ba7d04aa P c5b9dd725bb24fdca8d66cf909706f38: stopping tablet replica
I20260620 11:43:19.993070 17799 raft_consensus.cc:2229] T 8d70c36056d24c3fa6875034ba7d04aa P c5b9dd725bb24fdca8d66cf909706f38 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260620 11:43:19.993193 17799 raft_consensus.cc:2258] T 8d70c36056d24c3fa6875034ba7d04aa P c5b9dd725bb24fdca8d66cf909706f38 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 11:43:19.993435 17770 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 11:43:19.993293 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.234.193:42267 (local address 127.0.0.1:33356)
0620 11:43:19.993362 (+ 69us) negotiation.cc:107] Waiting for socket to connect
0620 11:43:19.993404 (+ 42us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.234.193:42267: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":47}
I20260620 11:43:19.993454 7083 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task1JRg7N/build/release/bin/kudu with pid 17494
W20260620 11:43:19.994330 17283 catalog_manager.cc:4584] TS b36a5c84137e48baaa4d49f2b9dcf6e8 (127.6.234.193:42267): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 8d70c36056d24c3fa6875034ba7d04aa: Network error: Client connection negotiation failed: client connection to 127.6.234.193:42267: connect: Connection refused (error 111)
I20260620 11:43:19.998332 17770 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 11:43:19.998204 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.234.194:41679 (local address 127.0.0.1:42676)
0620 11:43:19.998266 (+ 62us) negotiation.cc:107] Waiting for socket to connect
0620 11:43:19.998295 (+ 29us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.234.194:41679: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":42}
I20260620 11:43:19.998492 7083 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task1JRg7N/build/release/bin/kudu with pid 17628
I20260620 11:43:20.004640 7083 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task1JRg7N/build/release/bin/kudu with pid 17266
2026-06-20T11:43:20Z chronyd exiting
[ OK ] SecurityITest.TestRequireAuthenticationSecureCluster (1943 ms)
[ RUN ] SecurityITest.TestEncryptionWithKMSIntegration
Loading random data
Initializing database '/tmp/dist-test-task1JRg7N/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781955746769051-7083-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 20 11:43:20 dist-test-slave-cjc6 krb5kdc[17803](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 20 11:43:20 dist-test-slave-cjc6 krb5kdc[17803](info): set up 2 sockets
Jun 20 11:43:20 dist-test-slave-cjc6 krb5kdc[17803](info): commencing operation
krb5kdc: starting...
W20260620 11:43:21.038674 7083 mini_kdc.cc:121] Time spent starting KDC: real 1.014s user 0.000s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task1JRg7N/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781955746769051-7083-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task1JRg7N/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781955746769051-7083-0/krb5kdc/test-user.keytab.
Jun 20 11:43:21 dist-test-slave-cjc6 krb5kdc[17803](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781955801, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-20T11:43:21Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-20T11:43:21Z Disabled control of system clock
WARNING: no policy specified for rangeradmin/127.6.234.212@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "rangeradmin/127.6.234.212@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal rangeradmin/127.6.234.212@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task1JRg7N/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781955746769051-7083-0/krb5kdc/rangeradmin_127.6.234.212@KRBTEST.COM.keytab.
Entry for principal rangeradmin/127.6.234.212@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task1JRg7N/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781955746769051-7083-0/krb5kdc/rangeradmin_127.6.234.212@KRBTEST.COM.keytab.
WARNING: no policy specified for rangerlookup/127.6.234.212@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "rangerlookup/127.6.234.212@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal rangerlookup/127.6.234.212@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task1JRg7N/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781955746769051-7083-0/krb5kdc/rangerlookup_127.6.234.212@KRBTEST.COM.keytab.
Entry for principal rangerlookup/127.6.234.212@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task1JRg7N/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781955746769051-7083-0/krb5kdc/rangerlookup_127.6.234.212@KRBTEST.COM.keytab.
WARNING: no policy specified for HTTP/127.6.234.212@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "HTTP/127.6.234.212@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.6.234.212@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task1JRg7N/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781955746769051-7083-0/krb5kdc/HTTP_127.6.234.212@KRBTEST.COM.keytab.
Entry for principal HTTP/127.6.234.212@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task1JRg7N/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781955746769051-7083-0/krb5kdc/HTTP_127.6.234.212@KRBTEST.COM.keytab.
I20260620 11:43:21.194329 7083 mini_postgres.cc:62] Running initdb...
The files belonging to this database system will be owned by user "slave".
This user must also own the server process.
The database cluster will be initialized with locale "C".
The default database encoding has accordingly been set to "SQL_ASCII".
The default text search configuration will be set to "english".
Data page checksums are disabled.
creating directory /tmp/dist-test-task1JRg7N/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781955746769051-7083-0/minicluster-data/postgres ... ok
creating subdirectories ... ok
selecting dynamic shared memory implementation ... posix
selecting default max_connections ... 100
selecting default shared_buffers ... 128MB
selecting default time zone ... Etc/UTC
creating configuration files ... ok
running bootstrap script ... ok
performing post-bootstrap initialization ... ok
syncing data to disk ... ok
initdb: warning: enabling "trust" authentication for local connections
You can change this by editing pg_hba.conf or using the option -A, or
--auth-local and --auth-host, the next time you run initdb.
Success. You can now start the database server using:
/tmp/dist-test-task1JRg7N/build/release/bin/postgres/pg_ctl -D /tmp/dist-test-task1JRg7N/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781955746769051-7083-0/minicluster-data/postgres -l logfile start
2026-06-20 11:43:23.643 UTC [17833] LOG: starting PostgreSQL 12.2 on x86_64-pc-linux-gnu, compiled by cc (GCC) 8.3.1 20190311 (Red Hat 8.3.1-3), 64-bit
2026-06-20 11:43:23.643 UTC [17833] LOG: listening on IPv4 address "127.6.234.212", port 52871
2026-06-20 11:43:23.647 UTC [17833] LOG: listening on Unix socket "/tmp/.s.PGSQL.52871"
2026-06-20 11:43:23.681 UTC [17836] LOG: database system was shut down at 2026-06-20 11:43:22 UTC
2026-06-20 11:43:23.684 UTC [17833] LOG: database system is ready to accept connections
I20260620 11:43:24.626730 7083 mini_postgres.cc:96] Postgres bound to 52871
2026-06-20 11:43:24.632 UTC [17844] FATAL: database "slave" does not exist
127.6.234.212:52871 - accepting connections
I20260620 11:43:24.633409 7083 mini_ranger.cc:161] Starting Ranger...
I20260620 11:43:24.648298 7083 mini_ranger.cc:84] Created miniranger Postgres user
I20260620 11:43:25.112885 7083 mini_ranger.cc:87] Created ranger Postgres database
I20260620 11:43:25.112967 7083 mini_ranger.cc:178] Starting Ranger out of /tmp/dist-test-task1JRg7N/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781955746769051-7083-0/minicluster-data/ranger-admin
2026-06-20 11:43:25,388 [I] DB FLAVOR :POSTGRES
2026-06-20 11:43:25,388 [I] --------- Verifying Ranger DB connection ---------
2026-06-20 11:43:25,389 [I] Checking connection..
2026-06-20 11:43:25,389 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task1JRg7N/build/release/bin/postgresql.jar:/tmp/dist-test-task1JRg7N/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.6.234.212:52871/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select 1;"
2026-06-20 11:43:25,626 [I] Checking connection passed.
2026-06-20 11:43:25,627 [I] --------- Verifying version history table ---------
2026-06-20 11:43:25,627 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task1JRg7N/build/release/bin/postgresql.jar:/tmp/dist-test-task1JRg7N/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.6.234.212:52871/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_db_version_h') as temp;"
2026-06-20 11:43:25,872 [I] Table x_db_version_h does not exist in database ranger
2026-06-20 11:43:25,873 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task1JRg7N/build/release/bin/postgresql.jar:/tmp/dist-test-task1JRg7N/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.6.234.212:52871/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_db_version_h') as temp;"
2026-06-20 11:43:26,118 [I] Table x_db_version_h does not exist in database ranger
2026-06-20 11:43:26,119 [I] Importing x_db_version_h table schema to database ranger from file: create_dbversion_catalog.sql
2026-06-20 11:43:26,119 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task1JRg7N/build/release/bin/postgresql.jar:/tmp/dist-test-task1JRg7N/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.6.234.212:52871/ranger -u miniranger -p '********' -noheader -trim -c \; -input /tmp/dist-test-task1JRg7N/thirdparty/src/ranger-2.1.0-admin/db/postgres/create_dbversion_catalog.sql
2026-06-20 11:43:26.361 UTC [17908] WARNING: there is no transaction in progress
2026-06-20 11:43:26,367 [I] create_dbversion_catalog.sql file imported successfully
2026-06-20 11:43:26,367 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task1JRg7N/build/release/bin/postgresql.jar:/tmp/dist-test-task1JRg7N/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.6.234.212:52871/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_db_version_h') as temp;"
2026-06-20 11:43:26,608 [I] Table x_db_version_h already exists in database 'ranger'
2026-06-20 11:43:26,609 [I] --------- Importing Ranger Core DB Schema ---------
2026-06-20 11:43:26,609 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task1JRg7N/build/release/bin/postgresql.jar:/tmp/dist-test-task1JRg7N/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.6.234.212:52871/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'CORE_DB_SCHEMA' and active = 'Y';"
2026-06-20 11:43:26,863 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task1JRg7N/build/release/bin/postgresql.jar:/tmp/dist-test-task1JRg7N/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.6.234.212:52871/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'CORE_DB_SCHEMA' and active = 'N';"
2026-06-20 11:43:27,090 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task1JRg7N/build/release/bin/postgresql.jar:/tmp/dist-test-task1JRg7N/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.6.234.212:52871/ranger -u miniranger -p '********' -noheader -trim -c \; -query "insert into x_db_version_h (version, inst_at, inst_by, updated_at, updated_by,active) values ('CORE_DB_SCHEMA', current_timestamp, 'Ranger 2.1.0', current_timestamp, 'dist-test-slave-cjc6.c.gcp-upstream.internal','N') ;"
2026-06-20 11:43:27,328 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task1JRg7N/build/release/bin/postgresql.jar:/tmp/dist-test-task1JRg7N/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.6.234.212:52871/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_portal_user') as temp;"
2026-06-20 11:43:27,583 [I] Table x_portal_user does not exist in database ranger
2026-06-20 11:43:27,584 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task1JRg7N/build/release/bin/postgresql.jar:/tmp/dist-test-task1JRg7N/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.6.234.212:52871/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_policy_ref_group') as temp;"
2026-06-20 11:43:27,840 [I] Table x_policy_ref_group does not exist in database ranger
2026-06-20 11:43:27,840 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task1JRg7N/build/release/bin/postgresql.jar:/tmp/dist-test-task1JRg7N/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.6.234.212:52871/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'DB_PATCHES' and active = 'Y';"
2026-06-20 11:43:28,082 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task1JRg7N/build/release/bin/postgresql.jar:/tmp/dist-test-task1JRg7N/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.6.234.212:52871/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'JAVA_PATCHES' and active = 'Y';"
2026-06-20 11:43:28,313 [I] Importing DB schema to database ranger from file: ranger_core_db_postgres.sql
2026-06-20 11:43:28,314 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task1JRg7N/build/release/bin/postgresql.jar:/tmp/dist-test-task1JRg7N/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.6.234.212:52871/ranger -u miniranger -p '********' -noheader -trim -c \; -input /tmp/dist-test-task1JRg7N/thirdparty/src/ranger-2.1.0-admin/db/postgres/optimized/current/ranger_core_db_postgres.sql
2026-06-20 11:43:29.244 UTC [18016] WARNING: there is no transaction in progress
2026-06-20 11:43:29.260 UTC [18016] WARNING: there is no transaction in progress
2026-06-20 11:43:29.276 UTC [18016] WARNING: there is no transaction in progress
2026-06-20 11:43:29.291 UTC [18016] WARNING: there is no transaction in progress
2026-06-20 11:43:29.307 UTC [18016] WARNING: there is no transaction in progress
2026-06-20 11:43:29.323 UTC [18016] WARNING: there is no transaction in progress
2026-06-20 11:43:29.388 UTC [18016] WARNING: there is no transaction in progress
2026-06-20 11:43:29.394 UTC [18016] WARNING: there is no transaction in progress
2026-06-20 11:43:29.408 UTC [18016] WARNING: there is no transaction in progress
2026-06-20 11:43:29.418 UTC [18016] WARNING: there is no transaction in progress
2026-06-20 11:43:29.429 UTC [18016] WARNING: there is no transaction in progress
2026-06-20 11:43:29.442 UTC [18016] WARNING: there is no transaction in progress
2026-06-20 11:43:29.452 UTC [18016] WARNING: there is no transaction in progress
2026-06-20 11:43:29.458 UTC [18016] WARNING: there is no transaction in progress
2026-06-20 11:43:29.466 UTC [18016] WARNING: there is no transaction in progress
2026-06-20 11:43:29.473 UTC [18016] WARNING: there is no transaction in progress
2026-06-20 11:43:30,106 [I] ranger_core_db_postgres.sql file imported successfully
2026-06-20 11:43:30,106 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task1JRg7N/build/release/bin/postgresql.jar:/tmp/dist-test-task1JRg7N/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.6.234.212:52871/ranger -u miniranger -p '********' -noheader -trim -c \; -query "update x_db_version_h set inst_by='Ranger 2.1.0' where active='Y' and updated_by='localhost';"
2026-06-20 11:43:30,329 [I] Patches status entries updated from base ranger version to current installed ranger version:Ranger 2.1.0
2026-06-20 11:43:30,329 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task1JRg7N/build/release/bin/postgresql.jar:/tmp/dist-test-task1JRg7N/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.6.234.212:52871/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_portal_user') as temp;"
2026-06-20 11:43:30,593 [I] Table x_portal_user already exists in database 'ranger'
2026-06-20 11:43:30,594 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task1JRg7N/build/release/bin/postgresql.jar:/tmp/dist-test-task1JRg7N/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.6.234.212:52871/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_policy_ref_group') as temp;"
2026-06-20 11:43:30,826 [I] Table x_policy_ref_group already exists in database 'ranger'
2026-06-20 11:43:30,826 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task1JRg7N/build/release/bin/postgresql.jar:/tmp/dist-test-task1JRg7N/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.6.234.212:52871/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'DB_PATCHES' and active = 'Y';"
2026-06-20 11:43:31,070 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task1JRg7N/build/release/bin/postgresql.jar:/tmp/dist-test-task1JRg7N/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.6.234.212:52871/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'JAVA_PATCHES' and active = 'Y';"
2026-06-20 11:43:31,305 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task1JRg7N/build/release/bin/postgresql.jar:/tmp/dist-test-task1JRg7N/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.6.234.212:52871/ranger -u miniranger -p '********' -noheader -trim -c \; -query "update x_db_version_h set active='Y' where version='CORE_DB_SCHEMA' and active='N' and updated_by='dist-test-slave-cjc6.c.gcp-upstream.internal';"
2026-06-20 11:43:31,553 [I] CORE_DB_SCHEMA import status has been updated
2026-06-20 11:43:31,554 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task1JRg7N/build/release/bin/postgresql.jar:/tmp/dist-test-task1JRg7N/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.6.234.212:52871/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'DB_PATCHES' and inst_by = 'Ranger 2.1.0' and active = 'Y';"
2026-06-20 11:43:31,808 [I] DB_PATCHES have already been applied
I20260620 11:43:31.814039 7083 mini_ranger.cc:191] Using Ranger class path: /tmp/dist-test-task1JRg7N/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781955746769051-7083-0/minicluster-data/ranger-admin:/tmp/dist-test-task1JRg7N/thirdparty/src/ranger-2.1.0-admin/ews/lib/*:/usr/lib/jvm/java-1.8.0-openjdk-amd64/lib/*:/tmp/dist-test-task1JRg7N/thirdparty/src/hadoop-3.2.0/*:/tmp/dist-test-task1JRg7N/build/release/bin/postgresql.jar:/tmp/dist-test-task1JRg7N/thirdparty/src/ranger-2.1.0-admin/ews/webapp
I20260620 11:43:31.814105 7083 mini_ranger.cc:193] Using host: 127.6.234.212
I20260620 11:43:31.817058 7083 mini_ranger.cc:227] Ranger admin URL: http://127.6.234.212:51817
Jun 20, 2026 11:43:32 AM org.apache.ranger.server.tomcat.EmbeddedServer getKeyManagers
WARNING: Config 'ranger.keystore.file' or 'ranger.service.https.attrib.keystore.file' is not found or contains blank value
Jun 20, 2026 11:43:32 AM org.apache.ranger.server.tomcat.EmbeddedServer getTrustManagers
WARNING: Config 'ranger.truststore.file' is not found or contains blank value!
Jun 20, 2026 11:43:32 AM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Deriving webapp folder from catalina.base property. folder=/tmp/dist-test-task1JRg7N/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781955746769051-7083-0/minicluster-data/ranger-admin/ews/webapp
Jun 20, 2026 11:43:32 AM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Webapp file =/tmp/dist-test-task1JRg7N/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781955746769051-7083-0/minicluster-data/ranger-admin/ews/webapp, webAppName = /
Jun 20, 2026 11:43:32 AM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Adding webapp [/] = path [/tmp/dist-test-task1JRg7N/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781955746769051-7083-0/minicluster-data/ranger-admin/ews/webapp] .....
Jun 20, 2026 11:43:32 AM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Finished init of webapp [/] = path [/tmp/dist-test-task1JRg7N/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781955746769051-7083-0/minicluster-data/ranger-admin/ews/webapp].
Jun 20, 2026 11:43:32 AM org.apache.ranger.server.tomcat.EmbeddedServer startServer
INFO: Server Name : miniranger
I20260620 11:43:32.945422 7083 mini_ranger.cc:160] Time spent starting Ranger: real 8.312s user 0.000s sys 0.009s
Jun 20, 2026 11:43:52 AM com.sun.jersey.api.core.PackagesResourceConfig init
INFO: Scanning for root resource and provider classes in the packages:
org.apache.ranger.rest
org.apache.ranger.common
xa.rest
Jun 20, 2026 11:43:52 AM com.sun.jersey.api.core.ScanningResourceConfig logClasses
INFO: Root resource classes found:
class org.apache.ranger.rest.MetricsREST
class org.apache.ranger.rest.XKeyREST
class org.apache.ranger.rest.XAuditREST
class org.apache.ranger.rest.XUserREST
class org.apache.ranger.rest.UserREST
class org.apache.ranger.rest.ServiceREST
class org.apache.ranger.rest.PublicAPIs
class org.apache.ranger.rest.TagREST
class org.apache.ranger.rest.AssetREST
class org.apache.ranger.rest.RoleREST
class org.apache.ranger.rest.SecurityZoneREST
class org.apache.ranger.rest.PublicAPIsv2
Jun 20, 2026 11:43:52 AM com.sun.jersey.api.core.ScanningResourceConfig logClasses
INFO: Provider classes found:
class org.apache.ranger.common.RangerJAXBContextResolver
Jun 20, 2026 11:43:52 AM com.sun.jersey.spi.spring.container.servlet.SpringServlet getContext
INFO: Using default applicationContext
Jun 20, 2026 11:43:52 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, assetREST, of type org.apache.ranger.rest.AssetREST as a root resource class
Jun 20, 2026 11:43:52 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, metricsREST, of type org.apache.ranger.rest.MetricsREST as a root resource class
Jun 20, 2026 11:43:52 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, publicAPIs, of type org.apache.ranger.rest.PublicAPIs as a root resource class
Jun 20, 2026 11:43:52 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, publicAPIsv2, of type org.apache.ranger.rest.PublicAPIsv2 as a root resource class
Jun 20, 2026 11:43:52 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, roleREST, of type org.apache.ranger.rest.RoleREST as a root resource class
Jun 20, 2026 11:43:52 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, securityZoneREST, of type org.apache.ranger.rest.SecurityZoneREST as a root resource class
Jun 20, 2026 11:43:52 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, serviceREST, of type org.apache.ranger.rest.ServiceREST as a root resource class
Jun 20, 2026 11:43:52 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, tagREST, of type org.apache.ranger.rest.TagREST as a root resource class
Jun 20, 2026 11:43:52 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, userREST, of type org.apache.ranger.rest.UserREST as a root resource class
Jun 20, 2026 11:43:52 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, XAuditREST, of type org.apache.ranger.rest.XAuditREST as a root resource class
Jun 20, 2026 11:43:52 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, XKeyREST, of type org.apache.ranger.rest.XKeyREST as a root resource class
Jun 20, 2026 11:43:52 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, XUserREST, of type org.apache.ranger.rest.XUserREST as a root resource class
Jun 20, 2026 11:43:52 AM com.sun.jersey.server.impl.application.WebApplicationImpl _initiate
INFO: Initiating Jersey application, version 'Jersey: 1.19.3 10/24/2016 03:58 PM'
Jun 20, 2026 11:43:53 AM com.sun.jersey.spi.inject.Errors processErrorMessages
WARNING: The following warnings have been detected with resource and/or provider classes:
WARNING: A HTTP GET method, public void org.apache.ranger.rest.ServiceREST.getPoliciesInExcel(javax.servlet.http.HttpServletRequest,javax.servlet.http.HttpServletResponse), MUST return a non-void type.
WARNING: A HTTP GET method, public void org.apache.ranger.rest.ServiceREST.getPoliciesInJson(javax.servlet.http.HttpServletRequest,javax.servlet.http.HttpServletResponse,java.lang.Boolean), MUST return a non-void type.
WARNING: A HTTP GET method, public void org.apache.ranger.rest.ServiceREST.getPoliciesInCsv(javax.servlet.http.HttpServletRequest,javax.servlet.http.HttpServletResponse) throws java.io.IOException, MUST return a non-void type.
I20260620 11:43:54.197939 7083 mini_ranger.cc:261] Created Kudu service
WARNING: no policy specified for rangerkms/127.6.234.212@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "rangerkms/127.6.234.212@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal rangerkms/127.6.234.212@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task1JRg7N/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781955746769051-7083-0/krb5kdc/rangerkms_127.6.234.212@KRBTEST.COM.keytab.
Entry for principal rangerkms/127.6.234.212@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task1JRg7N/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781955746769051-7083-0/krb5kdc/rangerkms_127.6.234.212@KRBTEST.COM.keytab.
WARNING: no policy specified for HTTP/127.6.234.212@KRBTEST.COM; defaulting to no policy
add_principal: Principal or policy already exists while creating "HTTP/127.6.234.212@KRBTEST.COM".
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.6.234.212@KRBTEST.COM with kvno 3, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task1JRg7N/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781955746769051-7083-0/krb5kdc/HTTP_127.6.234.212@KRBTEST.COM.keytab.
Entry for principal HTTP/127.6.234.212@KRBTEST.COM with kvno 3, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task1JRg7N/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781955746769051-7083-0/krb5kdc/HTTP_127.6.234.212@KRBTEST.COM.keytab.
WARNING: no policy specified for keyadmin@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "keyadmin@KRBTEST.COM" created.
Jun 20 11:43:54 dist-test-slave-cjc6 krb5kdc[17803](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781955834, etypes {rep=17 tkt=17 ses=17}, keyadmin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for keyadmin@KRBTEST.COM:
I20260620 11:43:54.275799 7083 mini_ranger_kms.cc:208] Starting Ranger KMS...
I20260620 11:43:54.284411 7083 mini_ranger_kms.cc:78] Created minirangerkms Postgres user
I20260620 11:43:55.154979 7083 mini_ranger_kms.cc:81] Created rangerkms Postgres database
I20260620 11:43:55.155067 7083 mini_ranger_kms.cc:226] Starting Ranger KMS out of /tmp/dist-test-task1JRg7N/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781955746769051-7083-0/minicluster-data/ranger-kms
I20260620 11:43:55.155086 7083 mini_ranger_kms.cc:227] Using postgres at 127.6.234.212:52871
2026-06-20 11:43:55,331 [I] DB FLAVOR :POSTGRES
2026-06-20 11:43:55,331 [I] --------- Verifying Ranger DB connection ---------
2026-06-20 11:43:55,331 [I] Checking connection
2026-06-20 11:43:55,331 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task1JRg7N/build/release/bin/postgresql.jar:/tmp/dist-test-task1JRg7N/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781955746769051-7083-0/minicluster-data/ranger-kms/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.6.234.212:52871/rangerkms -u rangerkms -p '********' -noheader -trim -c \; -query "SELECT 1;"
2026-06-20 11:43:55,577 [I] connection success
2026-06-20 11:43:55,577 [I] --------- Verifying Ranger DB tables ---------
2026-06-20 11:43:55,577 [I] Verifying table ranger_masterkey in database rangerkms
2026-06-20 11:43:55,577 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task1JRg7N/build/release/bin/postgresql.jar:/tmp/dist-test-task1JRg7N/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781955746769051-7083-0/minicluster-data/ranger-kms/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.6.234.212:52871/rangerkms -u rangerkms -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='rangerkms' and table_name = 'ranger_masterkey') as temp;"
2026-06-20 11:43:55,802 [I] Table ranger_masterkey does not exist in database rangerkms
2026-06-20 11:43:55,803 [I] --------- Importing Ranger Core DB Schema ---------
2026-06-20 11:43:55,803 [I] Importing db schema to database rangerkms from file: kms_core_db_postgres.sql
2026-06-20 11:43:55,803 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task1JRg7N/build/release/bin/postgresql.jar:/tmp/dist-test-task1JRg7N/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781955746769051-7083-0/minicluster-data/ranger-kms/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.6.234.212:52871/rangerkms -u rangerkms -p '********' -noheader -trim -c \; -input /tmp/dist-test-task1JRg7N/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781955746769051-7083-0/minicluster-data/ranger-kms/db/postgres/kms_core_db_postgres.sql
2026-06-20 11:43:56,056 [I] kms_core_db_postgres.sql DB schema imported successfully
I20260620 11:43:56.466790 7083 mini_ranger_kms.cc:310] Created kms service
I20260620 11:43:56.629980 7083 mini_ranger_kms.cc:326] Created kudu user
I20260620 11:43:56.670411 7083 mini_ranger_kms.cc:343] Created rangerkms user
Jun 20 11:43:56 dist-test-slave-cjc6 krb5kdc[17803](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781955834, etypes {rep=17 tkt=17 ses=17}, keyadmin@KRBTEST.COM for HTTP/127.6.234.212@KRBTEST.COM
I20260620 11:43:57.066358 7083 mini_ranger_kms.cc:384] Added ranger policy
I20260620 11:43:57.066475 7083 mini_ranger_kms.cc:238] Using RangerKMS classpath: /tmp/dist-test-task1JRg7N/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781955746769051-7083-0/minicluster-data/ranger-kms:/tmp/dist-test-task1JRg7N/build/release/bin/postgresql.jar:/tmp/dist-test-task1JRg7N/thirdparty/src/ranger-2.1.0-kms/ews/webapp/WEB-INF/classes/lib/*:/tmp/dist-test-task1JRg7N/thirdparty/src/ranger-2.1.0-kms/ews/webapp/lib/*:/usr/lib/jvm/java-1.8.0-openjdk-amd64/lib/*
I20260620 11:43:57.066496 7083 mini_ranger_kms.cc:240] Using host: 127.6.234.212
I20260620 11:43:57.070139 7083 mini_ranger_kms.cc:276] Ranger KMS PID: 18196
I20260620 11:43:57.070251 7083 mini_ranger_kms.cc:277] Ranger KMS URL: http://127.6.234.212:57357
Jun 20, 2026 11:43:58 AM org.apache.ranger.server.tomcat.EmbeddedServer getKeyManagers
WARNING: Config 'ranger.keystore.file' or 'ranger.service.https.attrib.keystore.file' is not found or contains blank value
Jun 20, 2026 11:43:58 AM org.apache.ranger.server.tomcat.EmbeddedServer getTrustManagers
WARNING: Config 'ranger.truststore.file' is not found or contains blank value!
Jun 20, 2026 11:43:58 AM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Webapp file =/tmp/dist-test-task1JRg7N/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781955746769051-7083-0/minicluster-data/ranger-kms/ews/webapp, webAppName = /kms
Jun 20, 2026 11:43:58 AM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Adding webapp [/kms] = path [/tmp/dist-test-task1JRg7N/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781955746769051-7083-0/minicluster-data/ranger-kms/ews/webapp] .....
Jun 20, 2026 11:43:58 AM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Finished init of webapp [/kms] = path [/tmp/dist-test-task1JRg7N/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781955746769051-7083-0/minicluster-data/ranger-kms/ews/webapp].
Jun 20, 2026 11:43:58 AM org.apache.ranger.server.tomcat.EmbeddedServer startServer
INFO: Server Name : minirangerkms
I20260620 11:43:59.341854 7083 mini_ranger_kms.cc:207] Time spent starting Ranger KMS: real 5.066s user 0.000s sys 0.159s
I20260620 11:43:59.341959 7083 mini_ranger_kms.cc:397] {"name":"kuduclusterkey","cipher":"AES/CTR/NoPadding","length":128,"description":"kuduclusterkey"}
I20260620 11:43:59.342042 7083 mini_ranger_kms.cc:401] 127.6.234.212:57357/kms/v1/keys
Jun 20 11:43:59 dist-test-slave-cjc6 krb5kdc[17803](info): AS_REQ (1 etypes {17}) 127.0.0.1: ISSUE: authtime 1781955839, etypes {rep=17 tkt=17 ses=17}, rangerkms/127.6.234.212@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Jun 20 11:44:00 dist-test-slave-cjc6 krb5kdc[17803](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781955839, etypes {rep=17 tkt=17 ses=17}, rangerkms/127.6.234.212@KRBTEST.COM for HTTP/127.6.234.212@KRBTEST.COM
Jun 20 11:44:00 dist-test-slave-cjc6 krb5kdc[17803](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781955839, etypes {rep=17 tkt=17 ses=17}, rangerkms/127.6.234.212@KRBTEST.COM for HTTP/127.6.234.212@KRBTEST.COM
Jun 20 11:44:02 dist-test-slave-cjc6 krb5kdc[17803](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781955842, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
WARNING: no policy specified for kudu/127.6.234.254@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.6.234.254@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.6.234.254 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task1JRg7N/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781955746769051-7083-0/krb5kdc/kudu_127.6.234.254.keytab.
Entry for principal kudu/127.6.234.254 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task1JRg7N/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781955746769051-7083-0/krb5kdc/kudu_127.6.234.254.keytab.
I20260620 11:44:02.583954 7083 external_mini_cluster.cc:1326] Running /tmp/dist-test-task1JRg7N/build/release/bin/kudu
/tmp/dist-test-task1JRg7N/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task1JRg7N/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781955746769051-7083-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task1JRg7N/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781955746769051-7083-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task1JRg7N/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781955746769051-7083-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task1JRg7N/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781955746769051-7083-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.6.234.254:42195
--webserver_interface=127.6.234.254
--webserver_port=0
--keytab_file=/tmp/dist-test-task1JRg7N/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781955746769051-7083-0/krb5kdc/kudu_127.6.234.254.keytab
--principal=kudu/127.6.234.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.6.234.212:41343
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.6.234.254:42195
--ranger_config_path=/tmp/dist-test-task1JRg7N/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781955746769051-7083-0/minicluster-data/ranger-client
--trusted_user_acl=test-admin
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.6.234.212:57357/kms
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task1JRg7N/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781955746769051-7083-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task1JRg7N/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781955746769051-7083-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task1JRg7N/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781955746769051-7083-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 11:44:02.665159 18247 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 11:44:02.665376 18247 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 11:44:02.665398 18247 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 11:44:02.666649 18247 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 11:44:02.666693 18247 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 11:44:02.666708 18247 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 11:44:02.666720 18247 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 11:44:02.666733 18247 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 11:44:02.668246 18247 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.6.234.212:41343
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-task1JRg7N/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781955746769051-7083-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task1JRg7N/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781955746769051-7083-0/minicluster-data/master-0/wal
--ranger_kms_url=127.6.234.212:57357/kms
--trusted_user_acl=<redacted>
--ipki_ca_key_size=768
--master_addresses=127.6.234.254:42195
--ranger_config_path=/tmp/dist-test-task1JRg7N/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781955746769051-7083-0/minicluster-data/ranger-client
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task1JRg7N/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781955746769051-7083-0/krb5kdc/kudu_127.6.234.254.keytab
--principal=kudu/127.6.234.254
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.6.234.254:42195
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task1JRg7N/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781955746769051-7083-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.6.234.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.18247
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task1JRg7N/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781955746769051-7083-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 20 Jun 2026 10:32:37 UTC on ccycloud.quasar-bgyofv.root.comops.site
build id 4457173
I20260620 11:44:02.668530 18247 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 11:44:02.668746 18247 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 11:44:02.673472 18253 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 11:44:02.673492 18252 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 11:44:02.673480 18255 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 11:44:02.674130 18247 server_base.cc:1034] running on GCE node
I20260620 11:44:02.674391 18247 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 11:44:02.674667 18247 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 11:44:02.675828 18247 hybrid_clock.cc:648] HybridClock initialized: now 1781955842675808 us; error 38 us; skew 500 ppm
Jun 20 11:44:02 dist-test-slave-cjc6 krb5kdc[17803](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781955842, etypes {rep=17 tkt=17 ses=17}, kudu/127.6.234.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 11:44:02.678460 18247 init.cc:377] Logged in from keytab as kudu/127.6.234.254@KRBTEST.COM (short username kudu)
I20260620 11:44:02.678984 18247 webserver.cc:460] Webserver started at http://127.6.234.254:42793/ using document root <none> and password file <none>
I20260620 11:44:02.679219 18247 fs_manager.cc:359] Metadata directory not provided
I20260620 11:44:02.679270 18247 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 11:44:02.679389 18247 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Jun 20 11:44:02 dist-test-slave-cjc6 krb5kdc[17803](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781955842, etypes {rep=17 tkt=17 ses=17}, kudu/127.6.234.254@KRBTEST.COM for HTTP/127.6.234.212@KRBTEST.COM
I20260620 11:44:02.724354 18247 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task1JRg7N/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781955746769051-7083-0/minicluster-data/master-0/data/instance:
uuid: "062b1d98bcfd4f7d9ed182c3416809dd"
format_stamp: "Formatted at 2026-06-20 11:44:02 on dist-test-slave-cjc6"
server_key: "1ae11d7786df576269c821628a554e4e"
server_key_iv: "5d2f4e060186bcfd713cb3f281abddfa"
server_key_version: "kuduclusterkey@0"
I20260620 11:44:02.724833 18247 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task1JRg7N/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781955746769051-7083-0/minicluster-data/master-0/wal/instance:
uuid: "062b1d98bcfd4f7d9ed182c3416809dd"
format_stamp: "Formatted at 2026-06-20 11:44:02 on dist-test-slave-cjc6"
server_key: "1ae11d7786df576269c821628a554e4e"
server_key_iv: "5d2f4e060186bcfd713cb3f281abddfa"
server_key_version: "kuduclusterkey@0"
I20260620 11:44:02.726388 18247 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260620 11:44:02.740291 18262 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 11:44:02.740506 18247 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260620 11:44:02.740625 18247 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task1JRg7N/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781955746769051-7083-0/minicluster-data/master-0/data,/tmp/dist-test-task1JRg7N/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781955746769051-7083-0/minicluster-data/master-0/wal
uuid: "062b1d98bcfd4f7d9ed182c3416809dd"
format_stamp: "Formatted at 2026-06-20 11:44:02 on dist-test-slave-cjc6"
server_key: "1ae11d7786df576269c821628a554e4e"
server_key_iv: "5d2f4e060186bcfd713cb3f281abddfa"
server_key_version: "kuduclusterkey@0"
I20260620 11:44:02.740718 18247 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task1JRg7N/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781955746769051-7083-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task1JRg7N/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781955746769051-7083-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task1JRg7N/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781955746769051-7083-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 11:44:02.764848 18247 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 11:44:02.766848 18247 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 11:44:02.767016 18247 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 11:44:02.772068 18247 rpc_server.cc:307] RPC server started. Bound to: 127.6.234.254:42195
I20260620 11:44:02.772111 18314 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.6.234.254:42195 every 8 connection(s)
I20260620 11:44:02.772476 18247 server_base.cc:1166] Dumped server information to /tmp/dist-test-task1JRg7N/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781955746769051-7083-0/minicluster-data/master-0/data/info.pb
I20260620 11:44:02.773121 18315 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 11:44:02.775374 18315 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 062b1d98bcfd4f7d9ed182c3416809dd: Bootstrap starting.
I20260620 11:44:02.775928 18315 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 062b1d98bcfd4f7d9ed182c3416809dd: Neither blocks nor log segments found. Creating new log.
I20260620 11:44:02.776201 18315 log.cc:824] T 00000000000000000000000000000000 P 062b1d98bcfd4f7d9ed182c3416809dd: Log is configured to *not* fsync() on all Append() calls
I20260620 11:44:02.776850 18315 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 062b1d98bcfd4f7d9ed182c3416809dd: No bootstrap required, opened a new log
I20260620 11:44:02.778896 18315 raft_consensus.cc:348] T 00000000000000000000000000000000 P 062b1d98bcfd4f7d9ed182c3416809dd [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "062b1d98bcfd4f7d9ed182c3416809dd" member_type: VOTER last_known_addr { host: "127.6.234.254" port: 42195 } }
I20260620 11:44:02.779034 18315 raft_consensus.cc:374] T 00000000000000000000000000000000 P 062b1d98bcfd4f7d9ed182c3416809dd [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 11:44:02.779057 18315 raft_consensus.cc:729] T 00000000000000000000000000000000 P 062b1d98bcfd4f7d9ed182c3416809dd [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 062b1d98bcfd4f7d9ed182c3416809dd, State: Initialized, Role: FOLLOWER
I20260620 11:44:02.779201 18315 consensus_queue.cc:260] T 00000000000000000000000000000000 P 062b1d98bcfd4f7d9ed182c3416809dd [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "062b1d98bcfd4f7d9ed182c3416809dd" member_type: VOTER last_known_addr { host: "127.6.234.254" port: 42195 } }
I20260620 11:44:02.779275 18315 raft_consensus.cc:388] T 00000000000000000000000000000000 P 062b1d98bcfd4f7d9ed182c3416809dd [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260620 11:44:02.779299 18315 raft_consensus.cc:482] T 00000000000000000000000000000000 P 062b1d98bcfd4f7d9ed182c3416809dd [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260620 11:44:02.779364 18315 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 062b1d98bcfd4f7d9ed182c3416809dd [term 0 FOLLOWER]: Advancing to term 1
I20260620 11:44:02.779924 7083 external_mini_cluster.cc:1388] Started /tmp/dist-test-task1JRg7N/build/release/bin/kudu as pid 18247
I20260620 11:44:02.780005 18315 raft_consensus.cc:504] T 00000000000000000000000000000000 P 062b1d98bcfd4f7d9ed182c3416809dd [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "062b1d98bcfd4f7d9ed182c3416809dd" member_type: VOTER last_known_addr { host: "127.6.234.254" port: 42195 } }
I20260620 11:44:02.780045 7083 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task1JRg7N/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781955746769051-7083-0/minicluster-data/master-0/wal/instance
I20260620 11:44:02.780160 18315 leader_election.cc:302] T 00000000000000000000000000000000 P 062b1d98bcfd4f7d9ed182c3416809dd [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 062b1d98bcfd4f7d9ed182c3416809dd; no voters:
I20260620 11:44:02.780231 7083 external_mini_cluster.cc:1428] Setting key 30cb375dacf57d4843e20b48a07f6464
I20260620 11:44:02.780330 18315 leader_election.cc:288] T 00000000000000000000000000000000 P 062b1d98bcfd4f7d9ed182c3416809dd [CANDIDATE]: Term 1 election: Requested vote from peers
I20260620 11:44:02.780377 18318 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 062b1d98bcfd4f7d9ed182c3416809dd [term 1 FOLLOWER]: Leader election won for term 1
I20260620 11:44:02.780570 18315 sys_catalog.cc:564] T 00000000000000000000000000000000 P 062b1d98bcfd4f7d9ed182c3416809dd [sys.catalog]: configured and running, proceeding with master startup.
I20260620 11:44:02.780553 18318 raft_consensus.cc:686] T 00000000000000000000000000000000 P 062b1d98bcfd4f7d9ed182c3416809dd [term 1 LEADER]: Becoming Leader. State: Replica: 062b1d98bcfd4f7d9ed182c3416809dd, State: Running, Role: LEADER
I20260620 11:44:02.780715 18318 consensus_queue.cc:237] T 00000000000000000000000000000000 P 062b1d98bcfd4f7d9ed182c3416809dd [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "062b1d98bcfd4f7d9ed182c3416809dd" member_type: VOTER last_known_addr { host: "127.6.234.254" port: 42195 } }
I20260620 11:44:02.781028 18318 sys_catalog.cc:455] T 00000000000000000000000000000000 P 062b1d98bcfd4f7d9ed182c3416809dd [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "062b1d98bcfd4f7d9ed182c3416809dd" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "062b1d98bcfd4f7d9ed182c3416809dd" member_type: VOTER last_known_addr { host: "127.6.234.254" port: 42195 } } }
I20260620 11:44:02.781162 18319 sys_catalog.cc:455] T 00000000000000000000000000000000 P 062b1d98bcfd4f7d9ed182c3416809dd [sys.catalog]: SysCatalogTable state changed. Reason: New leader 062b1d98bcfd4f7d9ed182c3416809dd. Latest consensus state: current_term: 1 leader_uuid: "062b1d98bcfd4f7d9ed182c3416809dd" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "062b1d98bcfd4f7d9ed182c3416809dd" member_type: VOTER last_known_addr { host: "127.6.234.254" port: 42195 } } }
I20260620 11:44:02.781198 18318 sys_catalog.cc:458] T 00000000000000000000000000000000 P 062b1d98bcfd4f7d9ed182c3416809dd [sys.catalog]: This master's current role is: LEADER
I20260620 11:44:02.781461 18319 sys_catalog.cc:458] T 00000000000000000000000000000000 P 062b1d98bcfd4f7d9ed182c3416809dd [sys.catalog]: This master's current role is: LEADER
I20260620 11:44:02.782076 18329 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260620 11:44:02.782145 18315 ranger_client.cc:318] Using new properties file: /tmp/dist-test-task1JRg7N/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781955746769051-7083-0/minicluster-data/master-0/logs/kudu-ranger-subprocess-log4j2.properties
I20260620 11:44:02.782621 18329 catalog_manager.cc:1484] Initializing Kudu cluster ID...
Jun 20 11:44:02 dist-test-slave-cjc6 krb5kdc[17803](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781955842, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.6.234.254@KRBTEST.COM
I20260620 11:44:02.793414 18329 catalog_manager.cc:1347] Generated new cluster ID: 29f4541b09244e4ebbe3d84cbe700c3c
I20260620 11:44:02.793543 18329 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260620 11:44:02.797508 18322 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 11:44:02.780855 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48036 (local address 127.6.234.254:42195)
0620 11:44:02.781891 (+ 1036us) server_negotiation.cc:206] Beginning negotiation
0620 11:44:02.781901 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0620 11:44:02.781931 (+ 30us) server_negotiation.cc:407] Connection header received
0620 11:44:02.782044 (+ 113us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 11:44:02.782052 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 11:44:02.782109 (+ 57us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 11:44:02.782243 (+ 134us) server_negotiation.cc:226] Negotiated authn=SASL
0620 11:44:02.784151 (+ 1908us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 11:44:02.787129 (+ 2978us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 11:44:02.788418 (+ 1289us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 11:44:02.789112 (+ 694us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 11:44:02.789607 (+ 495us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 11:44:02.792306 (+ 2699us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 11:44:02.792311 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 11:44:02.792311 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 11:44:02.792325 (+ 14us) server_negotiation.cc:922] calling sasl_server_start()
0620 11:44:02.795274 (+ 2949us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 11:44:02.795696 (+ 422us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 11:44:02.795699 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 11:44:02.795699 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 11:44:02.795803 (+ 104us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 11:44:02.796039 (+ 236us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 11:44:02.796040 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 11:44:02.796041 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 11:44:02.796391 (+ 350us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 11:44:02.797016 (+ 625us) server_negotiation.cc:1035] Waiting for connection context
0620 11:44:02.797096 (+ 80us) server_negotiation.cc:299] Negotiation successful
0620 11:44:02.797221 (+ 125us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":995,"thread_start_us":73,"threads_started":1}
I20260620 11:44:02.823825 18329 catalog_manager.cc:1370] Generated new certificate authority record
I20260620 11:44:02.824702 18329 catalog_manager.cc:1504] Loading token signing keys...
I20260620 11:44:02.841562 18329 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 062b1d98bcfd4f7d9ed182c3416809dd: Generated new TSK 0
I20260620 11:44:02.841894 18329 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.6.234.193@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.6.234.193@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.6.234.193 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task1JRg7N/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781955746769051-7083-0/krb5kdc/kudu_127.6.234.193.keytab.
Entry for principal kudu/127.6.234.193 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task1JRg7N/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781955746769051-7083-0/krb5kdc/kudu_127.6.234.193.keytab.
I20260620 11:44:03.517400 7083 external_mini_cluster.cc:1326] Running /tmp/dist-test-task1JRg7N/build/release/bin/kudu
/tmp/dist-test-task1JRg7N/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task1JRg7N/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781955746769051-7083-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-task1JRg7N/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781955746769051-7083-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task1JRg7N/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781955746769051-7083-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-task1JRg7N/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781955746769051-7083-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.6.234.212:57357/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.6.234.193:0
--local_ip_for_outbound_sockets=127.6.234.193
--webserver_interface=127.6.234.193
--webserver_port=0
--tserver_master_addrs=127.6.234.254:42195
--keytab_file=/tmp/dist-test-task1JRg7N/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781955746769051-7083-0/krb5kdc/kudu_127.6.234.193.keytab
--principal=kudu/127.6.234.193
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.6.234.212:41343
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task1JRg7N/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781955746769051-7083-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task1JRg7N/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781955746769051-7083-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task1JRg7N/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781955746769051-7083-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 11:44:03.630681 18360 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 11:44:03.630960 18360 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 11:44:03.630999 18360 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 11:44:03.632779 18360 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 11:44:03.632856 18360 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 11:44:03.632902 18360 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.6.234.193
I20260620 11:44:03.634720 18360 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.6.234.212:41343
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-task1JRg7N/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781955746769051-7083-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-task1JRg7N/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781955746769051-7083-0/minicluster-data/ts-0/wal
--ranger_kms_url=127.6.234.212:57357/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task1JRg7N/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781955746769051-7083-0/krb5kdc/kudu_127.6.234.193.keytab
--principal=kudu/127.6.234.193
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.6.234.193:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task1JRg7N/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781955746769051-7083-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.6.234.193
--webserver_port=0
--tserver_master_addrs=127.6.234.254:42195
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.18360
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.6.234.193
--log_dir=/tmp/dist-test-task1JRg7N/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781955746769051-7083-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 20 Jun 2026 10:32:37 UTC on ccycloud.quasar-bgyofv.root.comops.site
build id 4457173
I20260620 11:44:03.634995 18360 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 11:44:03.635237 18360 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 11:44:03.640753 18367 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 11:44:03.640841 18368 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 11:44:03.640792 18370 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 11:44:03.641821 18360 server_base.cc:1034] running on GCE node
I20260620 11:44:03.642042 18360 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 11:44:03.642354 18360 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 11:44:03.643507 18360 hybrid_clock.cc:648] HybridClock initialized: now 1781955843643489 us; error 34 us; skew 500 ppm
Jun 20 11:44:03 dist-test-slave-cjc6 krb5kdc[17803](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781955843, etypes {rep=17 tkt=17 ses=17}, kudu/127.6.234.193@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 11:44:03.646436 18360 init.cc:377] Logged in from keytab as kudu/127.6.234.193@KRBTEST.COM (short username kudu)
I20260620 11:44:03.646886 18360 webserver.cc:460] Webserver started at http://127.6.234.193:38871/ using document root <none> and password file <none>
I20260620 11:44:03.647128 18360 fs_manager.cc:359] Metadata directory not provided
I20260620 11:44:03.647198 18360 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 11:44:03.647341 18360 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Jun 20 11:44:03 dist-test-slave-cjc6 krb5kdc[17803](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781955843, etypes {rep=17 tkt=17 ses=17}, kudu/127.6.234.193@KRBTEST.COM for HTTP/127.6.234.212@KRBTEST.COM
I20260620 11:44:03.658169 18360 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task1JRg7N/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781955746769051-7083-0/minicluster-data/ts-0/data/instance:
uuid: "034b5cfccdb0456c993c4b2889f2e762"
format_stamp: "Formatted at 2026-06-20 11:44:03 on dist-test-slave-cjc6"
server_key: "cb3fe44b9f6a75ac6ad8d676c7c024eb"
server_key_iv: "a195dca64572f91cf12703cf051a7585"
server_key_version: "kuduclusterkey@0"
I20260620 11:44:03.658675 18360 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task1JRg7N/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781955746769051-7083-0/minicluster-data/ts-0/wal/instance:
uuid: "034b5cfccdb0456c993c4b2889f2e762"
format_stamp: "Formatted at 2026-06-20 11:44:03 on dist-test-slave-cjc6"
server_key: "cb3fe44b9f6a75ac6ad8d676c7c024eb"
server_key_iv: "a195dca64572f91cf12703cf051a7585"
server_key_version: "kuduclusterkey@0"
I20260620 11:44:03.661005 18360 fs_manager.cc:691] Time spent creating directory manager: real 0.002s user 0.001s sys 0.000s
I20260620 11:44:03.677862 18377 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 11:44:03.678085 18360 fs_manager.cc:725] Time spent opening block manager: real 0.005s user 0.001s sys 0.000s
I20260620 11:44:03.678195 18360 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task1JRg7N/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781955746769051-7083-0/minicluster-data/ts-0/data,/tmp/dist-test-task1JRg7N/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781955746769051-7083-0/minicluster-data/ts-0/wal
uuid: "034b5cfccdb0456c993c4b2889f2e762"
format_stamp: "Formatted at 2026-06-20 11:44:03 on dist-test-slave-cjc6"
server_key: "cb3fe44b9f6a75ac6ad8d676c7c024eb"
server_key_iv: "a195dca64572f91cf12703cf051a7585"
server_key_version: "kuduclusterkey@0"
I20260620 11:44:03.678278 18360 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task1JRg7N/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781955746769051-7083-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-task1JRg7N/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781955746769051-7083-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-task1JRg7N/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781955746769051-7083-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 11:44:03.702636 18360 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 11:44:03.705123 18360 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 11:44:03.705319 18360 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 11:44:03.705622 18360 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 11:44:03.706091 18360 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 11:44:03.706189 18360 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 11:44:03.706286 18360 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 11:44:03.706357 18360 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 11:44:03.713018 18360 rpc_server.cc:307] RPC server started. Bound to: 127.6.234.193:39713
I20260620 11:44:03.713569 18360 server_base.cc:1166] Dumped server information to /tmp/dist-test-task1JRg7N/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781955746769051-7083-0/minicluster-data/ts-0/data/info.pb
I20260620 11:44:03.721158 18490 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.6.234.193:39713 every 8 connection(s)
I20260620 11:44:03.723623 7083 external_mini_cluster.cc:1388] Started /tmp/dist-test-task1JRg7N/build/release/bin/kudu as pid 18360
I20260620 11:44:03.723737 7083 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task1JRg7N/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781955746769051-7083-0/minicluster-data/ts-0/wal/instance
I20260620 11:44:03.723933 7083 external_mini_cluster.cc:1428] Setting key e115ce61b5405f8640f2fc5cedea0ec1
Jun 20 11:44:03 dist-test-slave-cjc6 krb5kdc[17803](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781955843, etypes {rep=17 tkt=17 ses=17}, kudu/127.6.234.193@KRBTEST.COM for kudu/127.6.234.254@KRBTEST.COM
I20260620 11:44:03.735406 18493 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 11:44:03.721938 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.234.254:42195 (local address 127.6.234.193:44189)
0620 11:44:03.722122 (+ 184us) negotiation.cc:107] Waiting for socket to connect
0620 11:44:03.722168 (+ 46us) client_negotiation.cc:174] Beginning negotiation
0620 11:44:03.722790 (+ 622us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 11:44:03.723221 (+ 431us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 11:44:03.723225 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 11:44:03.724256 (+ 1031us) client_negotiation.cc:189] Negotiated authn=SASL
0620 11:44:03.724531 (+ 275us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 11:44:03.724538 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 11:44:03.727125 (+ 2587us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 11:44:03.727126 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 11:44:03.728404 (+ 1278us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 11:44:03.728407 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 11:44:03.728970 (+ 563us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 11:44:03.728971 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 11:44:03.729113 (+ 142us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 11:44:03.729756 (+ 643us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 11:44:03.729767 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0620 11:44:03.731308 (+ 1541us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 11:44:03.733495 (+ 2187us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 11:44:03.733503 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 11:44:03.733506 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0620 11:44:03.733831 (+ 325us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 11:44:03.734129 (+ 298us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 11:44:03.734130 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 11:44:03.734130 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0620 11:44:03.734280 (+ 150us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 11:44:03.734610 (+ 330us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 11:44:03.734617 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 11:44:03.734816 (+ 199us) client_negotiation.cc:769] Sending connection context
0620 11:44:03.734928 (+ 112us) client_negotiation.cc:240] Negotiation successful
0620 11:44:03.735105 (+ 177us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":141,"thread_start_us":88,"threads_started":1}
I20260620 11:44:03.736303 18494 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 11:44:03.721951 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.6.234.193:44189 (local address 127.6.234.254:42195)
0620 11:44:03.722981 (+ 1030us) server_negotiation.cc:206] Beginning negotiation
0620 11:44:03.722983 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 11:44:03.722992 (+ 9us) server_negotiation.cc:407] Connection header received
0620 11:44:03.723018 (+ 26us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 11:44:03.723018 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 11:44:03.723032 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 11:44:03.723094 (+ 62us) server_negotiation.cc:226] Negotiated authn=SASL
0620 11:44:03.724632 (+ 1538us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 11:44:03.727008 (+ 2376us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 11:44:03.728505 (+ 1497us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 11:44:03.728830 (+ 325us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 11:44:03.728901 (+ 71us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 11:44:03.731427 (+ 2526us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 11:44:03.731430 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 11:44:03.731430 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 11:44:03.731438 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0620 11:44:03.733402 (+ 1964us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 11:44:03.733957 (+ 555us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 11:44:03.733958 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 11:44:03.733958 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 11:44:03.734026 (+ 68us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 11:44:03.734363 (+ 337us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 11:44:03.734364 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 11:44:03.734365 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 11:44:03.734525 (+ 160us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 11:44:03.736049 (+ 1524us) server_negotiation.cc:1035] Waiting for connection context
0620 11:44:03.736100 (+ 51us) server_negotiation.cc:299] Negotiation successful
0620 11:44:03.736197 (+ 97us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":999,"thread_start_us":87,"threads_started":1}
I20260620 11:44:03.737088 18491 heartbeater.cc:347] Connected to a master server at 127.6.234.254:42195
I20260620 11:44:03.737185 18491 heartbeater.cc:464] Registering TS with master...
I20260620 11:44:03.737375 18491 heartbeater.cc:511] Master 127.6.234.254:42195 requested a full tablet report, sending...
I20260620 11:44:03.737988 18279 ts_manager.cc:194] Registered new tserver with Master: 034b5cfccdb0456c993c4b2889f2e762 (127.6.234.193:39713)
WARNING: no policy specified for kudu/127.6.234.194@KRBTEST.COM; defaulting to no policy
I20260620 11:44:03.738916 18279 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.6.234.193@KRBTEST.COM'} at 127.6.234.193:44189
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.6.234.194@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.6.234.194 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task1JRg7N/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781955746769051-7083-0/krb5kdc/kudu_127.6.234.194.keytab.
Entry for principal kudu/127.6.234.194 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task1JRg7N/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781955746769051-7083-0/krb5kdc/kudu_127.6.234.194.keytab.
I20260620 11:44:03.758052 7083 external_mini_cluster.cc:1326] Running /tmp/dist-test-task1JRg7N/build/release/bin/kudu
/tmp/dist-test-task1JRg7N/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task1JRg7N/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781955746769051-7083-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-task1JRg7N/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781955746769051-7083-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task1JRg7N/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781955746769051-7083-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-task1JRg7N/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781955746769051-7083-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.6.234.212:57357/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.6.234.194:0
--local_ip_for_outbound_sockets=127.6.234.194
--webserver_interface=127.6.234.194
--webserver_port=0
--tserver_master_addrs=127.6.234.254:42195
--keytab_file=/tmp/dist-test-task1JRg7N/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781955746769051-7083-0/krb5kdc/kudu_127.6.234.194.keytab
--principal=kudu/127.6.234.194
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.6.234.212:41343
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task1JRg7N/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781955746769051-7083-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task1JRg7N/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781955746769051-7083-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task1JRg7N/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781955746769051-7083-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Jun 20 11:44:03 dist-test-slave-cjc6 krb5kdc[17803](info): AS_REQ (1 etypes {17}) 127.0.0.1: ISSUE: authtime 1781955843, etypes {rep=17 tkt=17 ses=17}, kudu/127.6.234.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
W20260620 11:44:03.852602 18499 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 11:44:03.852895 18499 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 11:44:03.852936 18499 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 11:44:03.854682 18499 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 11:44:03.854746 18499 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 11:44:03.854786 18499 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.6.234.194
I20260620 11:44:03.856874 18499 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.6.234.212:41343
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-task1JRg7N/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781955746769051-7083-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-task1JRg7N/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781955746769051-7083-0/minicluster-data/ts-1/wal
--ranger_kms_url=127.6.234.212:57357/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task1JRg7N/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781955746769051-7083-0/krb5kdc/kudu_127.6.234.194.keytab
--principal=kudu/127.6.234.194
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.6.234.194:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task1JRg7N/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781955746769051-7083-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.6.234.194
--webserver_port=0
--tserver_master_addrs=127.6.234.254:42195
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.18499
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.6.234.194
--log_dir=/tmp/dist-test-task1JRg7N/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781955746769051-7083-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 20 Jun 2026 10:32:37 UTC on ccycloud.quasar-bgyofv.root.comops.site
build id 4457173
I20260620 11:44:03.857156 18499 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 11:44:03.857402 18499 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 11:44:03.862735 18505 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 11:44:03.862848 18499 server_base.cc:1034] running on GCE node
W20260620 11:44:03.862735 18504 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 11:44:03.863065 18507 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 11:44:03.863373 18499 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 11:44:03.863626 18499 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 11:44:03.864785 18499 hybrid_clock.cc:648] HybridClock initialized: now 1781955843864772 us; error 26 us; skew 500 ppm
Jun 20 11:44:03 dist-test-slave-cjc6 krb5kdc[17803](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781955843, etypes {rep=17 tkt=17 ses=17}, kudu/127.6.234.194@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 11:44:03.867409 18499 init.cc:377] Logged in from keytab as kudu/127.6.234.194@KRBTEST.COM (short username kudu)
I20260620 11:44:03.867918 18499 webserver.cc:460] Webserver started at http://127.6.234.194:37359/ using document root <none> and password file <none>
I20260620 11:44:03.868157 18499 fs_manager.cc:359] Metadata directory not provided
I20260620 11:44:03.868228 18499 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 11:44:03.868386 18499 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Jun 20 11:44:03 dist-test-slave-cjc6 krb5kdc[17803](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781955843, etypes {rep=17 tkt=17 ses=17}, kudu/127.6.234.194@KRBTEST.COM for HTTP/127.6.234.212@KRBTEST.COM
I20260620 11:44:03.880887 18499 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task1JRg7N/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781955746769051-7083-0/minicluster-data/ts-1/data/instance:
uuid: "68a7e7d7ac884d258e13f6273dee1fc9"
format_stamp: "Formatted at 2026-06-20 11:44:03 on dist-test-slave-cjc6"
server_key: "1b41b08f4f9d17c648abdaedc31919e1"
server_key_iv: "4641eba56e07cf165bb60424489810b1"
server_key_version: "kuduclusterkey@0"
I20260620 11:44:03.881425 18499 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task1JRg7N/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781955746769051-7083-0/minicluster-data/ts-1/wal/instance:
uuid: "68a7e7d7ac884d258e13f6273dee1fc9"
format_stamp: "Formatted at 2026-06-20 11:44:03 on dist-test-slave-cjc6"
server_key: "1b41b08f4f9d17c648abdaedc31919e1"
server_key_iv: "4641eba56e07cf165bb60424489810b1"
server_key_version: "kuduclusterkey@0"
I20260620 11:44:03.883997 18499 fs_manager.cc:691] Time spent creating directory manager: real 0.002s user 0.002s sys 0.000s
I20260620 11:44:03.901990 18515 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 11:44:03.902287 18499 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260620 11:44:03.902447 18499 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task1JRg7N/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781955746769051-7083-0/minicluster-data/ts-1/data,/tmp/dist-test-task1JRg7N/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781955746769051-7083-0/minicluster-data/ts-1/wal
uuid: "68a7e7d7ac884d258e13f6273dee1fc9"
format_stamp: "Formatted at 2026-06-20 11:44:03 on dist-test-slave-cjc6"
server_key: "1b41b08f4f9d17c648abdaedc31919e1"
server_key_iv: "4641eba56e07cf165bb60424489810b1"
server_key_version: "kuduclusterkey@0"
I20260620 11:44:03.902627 18499 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task1JRg7N/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781955746769051-7083-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-task1JRg7N/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781955746769051-7083-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-task1JRg7N/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781955746769051-7083-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 11:44:03.919659 18499 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 11:44:03.922081 18499 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 11:44:03.922284 18499 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 11:44:03.922581 18499 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 11:44:03.923403 18499 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 11:44:03.923501 18499 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.001s
I20260620 11:44:03.923574 18499 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 11:44:03.923637 18499 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 11:44:03.933252 18499 rpc_server.cc:307] RPC server started. Bound to: 127.6.234.194:43959
I20260620 11:44:03.933836 18499 server_base.cc:1166] Dumped server information to /tmp/dist-test-task1JRg7N/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781955746769051-7083-0/minicluster-data/ts-1/data/info.pb
I20260620 11:44:03.937794 18628 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.6.234.194:43959 every 8 connection(s)
I20260620 11:44:03.940305 7083 external_mini_cluster.cc:1388] Started /tmp/dist-test-task1JRg7N/build/release/bin/kudu as pid 18499
I20260620 11:44:03.940424 7083 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task1JRg7N/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781955746769051-7083-0/minicluster-data/ts-1/wal/instance
I20260620 11:44:03.940613 7083 external_mini_cluster.cc:1428] Setting key 316b9aa565b73dec6281f0c7e93333cb
Jun 20 11:44:03 dist-test-slave-cjc6 krb5kdc[17803](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781955843, etypes {rep=17 tkt=17 ses=17}, kudu/127.6.234.194@KRBTEST.COM for kudu/127.6.234.254@KRBTEST.COM
WARNING: no policy specified for kudu/127.6.234.195@KRBTEST.COM; defaulting to no policy
I20260620 11:44:03.957494 18494 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 11:44:03.939793 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.6.234.194:53417 (local address 127.6.234.254:42195)
0620 11:44:03.939845 (+ 52us) server_negotiation.cc:206] Beginning negotiation
0620 11:44:03.939847 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 11:44:03.940103 (+ 256us) server_negotiation.cc:407] Connection header received
0620 11:44:03.940800 (+ 697us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 11:44:03.940801 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 11:44:03.940817 (+ 16us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 11:44:03.940854 (+ 37us) server_negotiation.cc:226] Negotiated authn=SASL
0620 11:44:03.941746 (+ 892us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 11:44:03.942498 (+ 752us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 11:44:03.944114 (+ 1616us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 11:44:03.944443 (+ 329us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 11:44:03.944512 (+ 69us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 11:44:03.953277 (+ 8765us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 11:44:03.953279 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 11:44:03.953280 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 11:44:03.953290 (+ 10us) server_negotiation.cc:922] calling sasl_server_start()
0620 11:44:03.955746 (+ 2456us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 11:44:03.956313 (+ 567us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 11:44:03.956314 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 11:44:03.956314 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 11:44:03.956371 (+ 57us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 11:44:03.956666 (+ 295us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 11:44:03.956667 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 11:44:03.956667 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 11:44:03.956827 (+ 160us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 11:44:03.956878 (+ 51us) server_negotiation.cc:1035] Waiting for connection context
0620 11:44:03.957289 (+ 411us) server_negotiation.cc:299] Negotiation successful
0620 11:44:03.957384 (+ 95us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":23}
I20260620 11:44:03.958338 18631 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 11:44:03.939760 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.234.254:42195 (local address 127.6.234.194:53417)
0620 11:44:03.939951 (+ 191us) negotiation.cc:107] Waiting for socket to connect
0620 11:44:03.940003 (+ 52us) client_negotiation.cc:174] Beginning negotiation
0620 11:44:03.940651 (+ 648us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 11:44:03.941054 (+ 403us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 11:44:03.941058 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 11:44:03.941308 (+ 250us) client_negotiation.cc:189] Negotiated authn=SASL
0620 11:44:03.941598 (+ 290us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 11:44:03.941605 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 11:44:03.942624 (+ 1019us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 11:44:03.942625 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 11:44:03.944026 (+ 1401us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 11:44:03.944029 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 11:44:03.944594 (+ 565us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 11:44:03.944595 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 11:44:03.944759 (+ 164us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 11:44:03.945963 (+ 1204us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 11:44:03.945973 (+ 10us) client_negotiation.cc:656] calling sasl_client_start()
0620 11:44:03.953120 (+ 7147us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 11:44:03.955854 (+ 2734us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 11:44:03.955862 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 11:44:03.955865 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0620 11:44:03.956216 (+ 351us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 11:44:03.956442 (+ 226us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 11:44:03.956442 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 11:44:03.956443 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0620 11:44:03.956576 (+ 133us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 11:44:03.956932 (+ 356us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 11:44:03.956939 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 11:44:03.957119 (+ 180us) client_negotiation.cc:769] Sending connection context
0620 11:44:03.957891 (+ 772us) client_negotiation.cc:240] Negotiation successful
0620 11:44:03.958070 (+ 179us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":140,"thread_start_us":102,"threads_started":1}
I20260620 11:44:03.959048 18629 heartbeater.cc:347] Connected to a master server at 127.6.234.254:42195
I20260620 11:44:03.959151 18629 heartbeater.cc:464] Registering TS with master...
I20260620 11:44:03.959342 18629 heartbeater.cc:511] Master 127.6.234.254:42195 requested a full tablet report, sending...
I20260620 11:44:03.959780 18279 ts_manager.cc:194] Registered new tserver with Master: 68a7e7d7ac884d258e13f6273dee1fc9 (127.6.234.194:43959)
I20260620 11:44:03.960273 18279 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.6.234.194@KRBTEST.COM'} at 127.6.234.194:53417
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.6.234.195@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.6.234.195 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task1JRg7N/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781955746769051-7083-0/krb5kdc/kudu_127.6.234.195.keytab.
Entry for principal kudu/127.6.234.195 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task1JRg7N/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781955746769051-7083-0/krb5kdc/kudu_127.6.234.195.keytab.
I20260620 11:44:03.988620 7083 external_mini_cluster.cc:1326] Running /tmp/dist-test-task1JRg7N/build/release/bin/kudu
/tmp/dist-test-task1JRg7N/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task1JRg7N/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781955746769051-7083-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-task1JRg7N/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781955746769051-7083-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task1JRg7N/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781955746769051-7083-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-task1JRg7N/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781955746769051-7083-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.6.234.212:57357/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.6.234.195:0
--local_ip_for_outbound_sockets=127.6.234.195
--webserver_interface=127.6.234.195
--webserver_port=0
--tserver_master_addrs=127.6.234.254:42195
--keytab_file=/tmp/dist-test-task1JRg7N/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781955746769051-7083-0/krb5kdc/kudu_127.6.234.195.keytab
--principal=kudu/127.6.234.195
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.6.234.212:41343
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task1JRg7N/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781955746769051-7083-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task1JRg7N/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781955746769051-7083-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task1JRg7N/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781955746769051-7083-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 11:44:04.123684 18634 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 11:44:04.123911 18634 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 11:44:04.123947 18634 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 11:44:04.125161 18634 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 11:44:04.125231 18634 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 11:44:04.125274 18634 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.6.234.195
I20260620 11:44:04.126778 18634 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.6.234.212:41343
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-task1JRg7N/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781955746769051-7083-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-task1JRg7N/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781955746769051-7083-0/minicluster-data/ts-2/wal
--ranger_kms_url=127.6.234.212:57357/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task1JRg7N/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781955746769051-7083-0/krb5kdc/kudu_127.6.234.195.keytab
--principal=kudu/127.6.234.195
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.6.234.195:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task1JRg7N/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781955746769051-7083-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.6.234.195
--webserver_port=0
--tserver_master_addrs=127.6.234.254:42195
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.18634
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.6.234.195
--log_dir=/tmp/dist-test-task1JRg7N/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781955746769051-7083-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 20 Jun 2026 10:32:37 UTC on ccycloud.quasar-bgyofv.root.comops.site
build id 4457173
I20260620 11:44:04.127028 18634 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 11:44:04.127274 18634 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 11:44:04.132155 18640 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 11:44:04.132160 18642 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 11:44:04.132534 18639 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 11:44:04.133208 18634 server_base.cc:1034] running on GCE node
I20260620 11:44:04.133412 18634 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 11:44:04.133653 18634 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 11:44:04.136924 18634 hybrid_clock.cc:648] HybridClock initialized: now 1781955844136901 us; error 32 us; skew 500 ppm
Jun 20 11:44:04 dist-test-slave-cjc6 krb5kdc[17803](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781955844, etypes {rep=17 tkt=17 ses=17}, kudu/127.6.234.195@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 11:44:04.143635 18634 init.cc:377] Logged in from keytab as kudu/127.6.234.195@KRBTEST.COM (short username kudu)
I20260620 11:44:04.144174 18634 webserver.cc:460] Webserver started at http://127.6.234.195:37841/ using document root <none> and password file <none>
I20260620 11:44:04.144440 18634 fs_manager.cc:359] Metadata directory not provided
I20260620 11:44:04.144493 18634 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 11:44:04.144616 18634 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Jun 20 11:44:04 dist-test-slave-cjc6 krb5kdc[17803](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781955844, etypes {rep=17 tkt=17 ses=17}, kudu/127.6.234.195@KRBTEST.COM for HTTP/127.6.234.212@KRBTEST.COM
I20260620 11:44:04.160472 18634 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task1JRg7N/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781955746769051-7083-0/minicluster-data/ts-2/data/instance:
uuid: "1e4d0bc08a854ddf8b3b9e8d5dafd21f"
format_stamp: "Formatted at 2026-06-20 11:44:04 on dist-test-slave-cjc6"
server_key: "7ea6f75b37b6aa59e336af18e285bb38"
server_key_iv: "a18ee4c79e1b15987576841a33860612"
server_key_version: "kuduclusterkey@0"
I20260620 11:44:04.161013 18634 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task1JRg7N/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781955746769051-7083-0/minicluster-data/ts-2/wal/instance:
uuid: "1e4d0bc08a854ddf8b3b9e8d5dafd21f"
format_stamp: "Formatted at 2026-06-20 11:44:04 on dist-test-slave-cjc6"
server_key: "7ea6f75b37b6aa59e336af18e285bb38"
server_key_iv: "a18ee4c79e1b15987576841a33860612"
server_key_version: "kuduclusterkey@0"
I20260620 11:44:04.166109 18634 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.001s sys 0.000s
I20260620 11:44:04.202373 18649 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 11:44:04.202654 18634 fs_manager.cc:725] Time spent opening block manager: real 0.012s user 0.001s sys 0.000s
I20260620 11:44:04.202767 18634 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task1JRg7N/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781955746769051-7083-0/minicluster-data/ts-2/data,/tmp/dist-test-task1JRg7N/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781955746769051-7083-0/minicluster-data/ts-2/wal
uuid: "1e4d0bc08a854ddf8b3b9e8d5dafd21f"
format_stamp: "Formatted at 2026-06-20 11:44:04 on dist-test-slave-cjc6"
server_key: "7ea6f75b37b6aa59e336af18e285bb38"
server_key_iv: "a18ee4c79e1b15987576841a33860612"
server_key_version: "kuduclusterkey@0"
I20260620 11:44:04.202865 18634 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task1JRg7N/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781955746769051-7083-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-task1JRg7N/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781955746769051-7083-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-task1JRg7N/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781955746769051-7083-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 11:44:04.229907 18634 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 11:44:04.232446 18634 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 11:44:04.232645 18634 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 11:44:04.232903 18634 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 11:44:04.233325 18634 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 11:44:04.233418 18634 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 11:44:04.233495 18634 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 11:44:04.233556 18634 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 11:44:04.240383 18634 rpc_server.cc:307] RPC server started. Bound to: 127.6.234.195:33419
I20260620 11:44:04.240935 18634 server_base.cc:1166] Dumped server information to /tmp/dist-test-task1JRg7N/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781955746769051-7083-0/minicluster-data/ts-2/data/info.pb
I20260620 11:44:04.244145 18762 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.6.234.195:33419 every 8 connection(s)
I20260620 11:44:04.247458 7083 external_mini_cluster.cc:1388] Started /tmp/dist-test-task1JRg7N/build/release/bin/kudu as pid 18634
I20260620 11:44:04.247571 7083 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task1JRg7N/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781955746769051-7083-0/minicluster-data/ts-2/wal/instance
I20260620 11:44:04.247749 7083 external_mini_cluster.cc:1428] Setting key 548cdd711d9c8073c91c8532c8af9112
Jun 20 11:44:04 dist-test-slave-cjc6 krb5kdc[17803](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781955844, etypes {rep=17 tkt=17 ses=17}, kudu/127.6.234.195@KRBTEST.COM for kudu/127.6.234.254@KRBTEST.COM
I20260620 11:44:04.256536 18494 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 11:44:04.244987 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.6.234.195:56543 (local address 127.6.234.254:42195)
0620 11:44:04.245029 (+ 42us) server_negotiation.cc:206] Beginning negotiation
0620 11:44:04.245031 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 11:44:04.245556 (+ 525us) server_negotiation.cc:407] Connection header received
0620 11:44:04.246301 (+ 745us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 11:44:04.246302 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 11:44:04.246314 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 11:44:04.246345 (+ 31us) server_negotiation.cc:226] Negotiated authn=SASL
0620 11:44:04.247215 (+ 870us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 11:44:04.247976 (+ 761us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 11:44:04.250261 (+ 2285us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 11:44:04.250547 (+ 286us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 11:44:04.250593 (+ 46us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 11:44:04.253195 (+ 2602us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 11:44:04.253196 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 11:44:04.253196 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 11:44:04.253204 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0620 11:44:04.254943 (+ 1739us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 11:44:04.255449 (+ 506us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 11:44:04.255450 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 11:44:04.255450 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 11:44:04.255509 (+ 59us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 11:44:04.255801 (+ 292us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 11:44:04.255802 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 11:44:04.255803 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 11:44:04.255934 (+ 131us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 11:44:04.255976 (+ 42us) server_negotiation.cc:1035] Waiting for connection context
0620 11:44:04.256354 (+ 378us) server_negotiation.cc:299] Negotiation successful
0620 11:44:04.256440 (+ 86us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":18}
I20260620 11:44:04.257287 18765 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 11:44:04.244711 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.234.254:42195 (local address 127.6.234.195:56543)
0620 11:44:04.245320 (+ 609us) negotiation.cc:107] Waiting for socket to connect
0620 11:44:04.245469 (+ 149us) client_negotiation.cc:174] Beginning negotiation
0620 11:44:04.246159 (+ 690us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 11:44:04.246546 (+ 387us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 11:44:04.246550 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 11:44:04.246804 (+ 254us) client_negotiation.cc:189] Negotiated authn=SASL
0620 11:44:04.247084 (+ 280us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 11:44:04.247090 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 11:44:04.248747 (+ 1657us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 11:44:04.248748 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 11:44:04.250123 (+ 1375us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 11:44:04.250126 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 11:44:04.250690 (+ 564us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 11:44:04.250691 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 11:44:04.250825 (+ 134us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 11:44:04.251447 (+ 622us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 11:44:04.251457 (+ 10us) client_negotiation.cc:656] calling sasl_client_start()
0620 11:44:04.253073 (+ 1616us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 11:44:04.255058 (+ 1985us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 11:44:04.255064 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 11:44:04.255066 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0620 11:44:04.255355 (+ 289us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 11:44:04.255584 (+ 229us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 11:44:04.255585 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 11:44:04.255585 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0620 11:44:04.255716 (+ 131us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 11:44:04.256022 (+ 306us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 11:44:04.256027 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 11:44:04.256191 (+ 164us) client_negotiation.cc:769] Sending connection context
0620 11:44:04.256890 (+ 699us) client_negotiation.cc:240] Negotiation successful
0620 11:44:04.257046 (+ 156us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":563,"thread_start_us":84,"threads_started":1}
I20260620 11:44:04.257977 18763 heartbeater.cc:347] Connected to a master server at 127.6.234.254:42195
I20260620 11:44:04.258069 18763 heartbeater.cc:464] Registering TS with master...
I20260620 11:44:04.258282 18763 heartbeater.cc:511] Master 127.6.234.254:42195 requested a full tablet report, sending...
I20260620 11:44:04.258692 18279 ts_manager.cc:194] Registered new tserver with Master: 1e4d0bc08a854ddf8b3b9e8d5dafd21f (127.6.234.195:33419)
I20260620 11:44:04.259195 18279 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.6.234.195@KRBTEST.COM'} at 127.6.234.195:56543
I20260620 11:44:04.260484 7083 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260620 11:44:04.279322 18494 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 11:44:04.265965 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48042 (local address 127.6.234.254:42195)
0620 11:44:04.266004 (+ 39us) server_negotiation.cc:206] Beginning negotiation
0620 11:44:04.266006 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 11:44:04.269874 (+ 3868us) server_negotiation.cc:407] Connection header received
0620 11:44:04.269977 (+ 103us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 11:44:04.269978 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 11:44:04.269989 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 11:44:04.270019 (+ 30us) server_negotiation.cc:226] Negotiated authn=SASL
0620 11:44:04.271152 (+ 1133us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 11:44:04.271919 (+ 767us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 11:44:04.273015 (+ 1096us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 11:44:04.273306 (+ 291us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 11:44:04.273348 (+ 42us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 11:44:04.274988 (+ 1640us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 11:44:04.274989 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 11:44:04.274990 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 11:44:04.274996 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0620 11:44:04.277326 (+ 2330us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 11:44:04.278434 (+ 1108us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 11:44:04.278436 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 11:44:04.278436 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 11:44:04.278496 (+ 60us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 11:44:04.278733 (+ 237us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 11:44:04.278734 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 11:44:04.278735 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 11:44:04.278865 (+ 130us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 11:44:04.278946 (+ 81us) server_negotiation.cc:1035] Waiting for connection context
0620 11:44:04.279148 (+ 202us) server_negotiation.cc:299] Negotiation successful
0620 11:44:04.279225 (+ 77us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":19}
I20260620 11:44:04.281594 18279 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:48042:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260620 11:44:04.283931 18279 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260620 11:44:04.304615 18775 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 11:44:04.297129 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.234.193:39713 (local address 127.0.0.1:53574)
0620 11:44:04.298537 (+ 1408us) negotiation.cc:107] Waiting for socket to connect
0620 11:44:04.298582 (+ 45us) client_negotiation.cc:174] Beginning negotiation
0620 11:44:04.298651 (+ 69us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 11:44:04.299102 (+ 451us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 11:44:04.299106 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 11:44:04.299108 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 11:44:04.299205 (+ 97us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 11:44:04.299207 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 11:44:04.302075 (+ 2868us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 11:44:04.302076 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 11:44:04.303399 (+ 1323us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 11:44:04.303402 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 11:44:04.304338 (+ 936us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 11:44:04.304339 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 11:44:04.304425 (+ 86us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 11:44:04.304439 (+ 14us) client_negotiation.cc:769] Sending connection context
0620 11:44:04.304486 (+ 47us) client_negotiation.cc:240] Negotiation successful
0620 11:44:04.304533 (+ 47us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1378,"thread_start_us":98,"threads_started":1}
I20260620 11:44:04.305400 18778 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 11:44:04.297945 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53574 (local address 127.6.234.193:39713)
0620 11:44:04.298913 (+ 968us) server_negotiation.cc:206] Beginning negotiation
0620 11:44:04.298918 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 11:44:04.298936 (+ 18us) server_negotiation.cc:407] Connection header received
0620 11:44:04.298972 (+ 36us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 11:44:04.298974 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 11:44:04.298992 (+ 18us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 11:44:04.299293 (+ 301us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 11:44:04.299353 (+ 60us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 11:44:04.300413 (+ 1060us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 11:44:04.303506 (+ 3093us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 11:44:04.304249 (+ 743us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 11:44:04.305184 (+ 935us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 11:44:04.305258 (+ 74us) server_negotiation.cc:1035] Waiting for connection context
0620 11:44:04.305284 (+ 26us) server_negotiation.cc:299] Negotiation successful
0620 11:44:04.305326 (+ 42us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":935,"thread_start_us":60,"threads_started":1}
I20260620 11:44:04.307072 18415 tablet_service.cc:1467] Processing CreateTablet for tablet 1601a6a976144e03b58a9db19004c145 (DEFAULT_TABLE table=test-table [id=7eeadee59b704a818724666f04ea2d0a]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 11:44:04.307488 18415 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 1601a6a976144e03b58a9db19004c145. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 11:44:04.308245 18776 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 11:44:04.297395 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.234.195:33419 (local address 127.0.0.1:60312)
0620 11:44:04.298757 (+ 1362us) negotiation.cc:107] Waiting for socket to connect
0620 11:44:04.298765 (+ 8us) client_negotiation.cc:174] Beginning negotiation
0620 11:44:04.298798 (+ 33us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 11:44:04.300714 (+ 1916us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 11:44:04.300715 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 11:44:04.300717 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 11:44:04.300766 (+ 49us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 11:44:04.300767 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 11:44:04.302712 (+ 1945us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 11:44:04.302713 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 11:44:04.304020 (+ 1307us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 11:44:04.304022 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 11:44:04.308025 (+ 4003us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 11:44:04.308026 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 11:44:04.308090 (+ 64us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 11:44:04.308095 (+ 5us) client_negotiation.cc:769] Sending connection context
0620 11:44:04.308127 (+ 32us) client_negotiation.cc:240] Negotiation successful
0620 11:44:04.308165 (+ 38us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1347,"thread_start_us":68,"threads_started":1}
I20260620 11:44:04.308624 18779 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 11:44:04.298123 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60312 (local address 127.6.234.195:33419)
0620 11:44:04.300546 (+ 2423us) server_negotiation.cc:206] Beginning negotiation
0620 11:44:04.300551 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 11:44:04.300560 (+ 9us) server_negotiation.cc:407] Connection header received
0620 11:44:04.300596 (+ 36us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 11:44:04.300598 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 11:44:04.300614 (+ 16us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 11:44:04.300838 (+ 224us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 11:44:04.300917 (+ 79us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 11:44:04.301966 (+ 1049us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 11:44:04.304131 (+ 2165us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 11:44:04.304865 (+ 734us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 11:44:04.308349 (+ 3484us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 11:44:04.308434 (+ 85us) server_negotiation.cc:1035] Waiting for connection context
0620 11:44:04.308509 (+ 75us) server_negotiation.cc:299] Negotiation successful
0620 11:44:04.308549 (+ 40us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":2388,"thread_start_us":92,"threads_started":1}
I20260620 11:44:04.310765 18697 tablet_service.cc:1467] Processing CreateTablet for tablet 1601a6a976144e03b58a9db19004c145 (DEFAULT_TABLE table=test-table [id=7eeadee59b704a818724666f04ea2d0a]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 11:44:04.311094 18697 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 1601a6a976144e03b58a9db19004c145. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 11:44:04.319345 18775 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 11:44:04.297563 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.234.194:43959 (local address 127.0.0.1:52632)
0620 11:44:04.305047 (+ 7484us) negotiation.cc:107] Waiting for socket to connect
0620 11:44:04.305056 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0620 11:44:04.305093 (+ 37us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 11:44:04.305806 (+ 713us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 11:44:04.305807 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 11:44:04.305809 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 11:44:04.305837 (+ 28us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 11:44:04.305839 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 11:44:04.316953 (+ 11114us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 11:44:04.316955 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 11:44:04.318069 (+ 1114us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 11:44:04.318072 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 11:44:04.319107 (+ 1035us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 11:44:04.319109 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 11:44:04.319180 (+ 71us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 11:44:04.319184 (+ 4us) client_negotiation.cc:769] Sending connection context
0620 11:44:04.319215 (+ 31us) client_negotiation.cc:240] Negotiation successful
0620 11:44:04.319256 (+ 41us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":7474,"thread_start_us":56,"threads_started":1}
I20260620 11:44:04.319911 18780 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 11:44:04.298309 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52632 (local address 127.6.234.194:43959)
0620 11:44:04.302658 (+ 4349us) server_negotiation.cc:206] Beginning negotiation
0620 11:44:04.302662 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 11:44:04.305612 (+ 2950us) server_negotiation.cc:407] Connection header received
0620 11:44:04.305660 (+ 48us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 11:44:04.305663 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 11:44:04.305679 (+ 16us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 11:44:04.305908 (+ 229us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 11:44:04.305985 (+ 77us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 11:44:04.316848 (+ 10863us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 11:44:04.318249 (+ 1401us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 11:44:04.319009 (+ 760us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 11:44:04.319685 (+ 676us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 11:44:04.319770 (+ 85us) server_negotiation.cc:1035] Waiting for connection context
0620 11:44:04.319797 (+ 27us) server_negotiation.cc:299] Negotiation successful
0620 11:44:04.319836 (+ 39us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":4313,"thread_start_us":75,"threads_started":1}
I20260620 11:44:04.320673 18563 tablet_service.cc:1467] Processing CreateTablet for tablet 1601a6a976144e03b58a9db19004c145 (DEFAULT_TABLE table=test-table [id=7eeadee59b704a818724666f04ea2d0a]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 11:44:04.321012 18563 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 1601a6a976144e03b58a9db19004c145. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 11:44:04.321837 18781 tablet_bootstrap.cc:492] T 1601a6a976144e03b58a9db19004c145 P 034b5cfccdb0456c993c4b2889f2e762: Bootstrap starting.
I20260620 11:44:04.322526 18781 tablet_bootstrap.cc:654] T 1601a6a976144e03b58a9db19004c145 P 034b5cfccdb0456c993c4b2889f2e762: Neither blocks nor log segments found. Creating new log.
I20260620 11:44:04.323174 18781 log.cc:824] T 1601a6a976144e03b58a9db19004c145 P 034b5cfccdb0456c993c4b2889f2e762: Log is configured to *not* fsync() on all Append() calls
I20260620 11:44:04.323506 18782 tablet_bootstrap.cc:492] T 1601a6a976144e03b58a9db19004c145 P 1e4d0bc08a854ddf8b3b9e8d5dafd21f: Bootstrap starting.
I20260620 11:44:04.324394 18782 tablet_bootstrap.cc:654] T 1601a6a976144e03b58a9db19004c145 P 1e4d0bc08a854ddf8b3b9e8d5dafd21f: Neither blocks nor log segments found. Creating new log.
I20260620 11:44:04.324651 18782 log.cc:824] T 1601a6a976144e03b58a9db19004c145 P 1e4d0bc08a854ddf8b3b9e8d5dafd21f: Log is configured to *not* fsync() on all Append() calls
I20260620 11:44:04.325891 18781 tablet_bootstrap.cc:492] T 1601a6a976144e03b58a9db19004c145 P 034b5cfccdb0456c993c4b2889f2e762: No bootstrap required, opened a new log
I20260620 11:44:04.326010 18781 ts_tablet_manager.cc:1397] T 1601a6a976144e03b58a9db19004c145 P 034b5cfccdb0456c993c4b2889f2e762: Time spent bootstrapping tablet: real 0.004s user 0.002s sys 0.000s
I20260620 11:44:04.328557 18781 raft_consensus.cc:348] T 1601a6a976144e03b58a9db19004c145 P 034b5cfccdb0456c993c4b2889f2e762 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "034b5cfccdb0456c993c4b2889f2e762" member_type: VOTER last_known_addr { host: "127.6.234.193" port: 39713 } } peers { permanent_uuid: "1e4d0bc08a854ddf8b3b9e8d5dafd21f" member_type: VOTER last_known_addr { host: "127.6.234.195" port: 33419 } } peers { permanent_uuid: "68a7e7d7ac884d258e13f6273dee1fc9" member_type: VOTER last_known_addr { host: "127.6.234.194" port: 43959 } }
I20260620 11:44:04.329087 18781 raft_consensus.cc:374] T 1601a6a976144e03b58a9db19004c145 P 034b5cfccdb0456c993c4b2889f2e762 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 11:44:04.329208 18781 raft_consensus.cc:729] T 1601a6a976144e03b58a9db19004c145 P 034b5cfccdb0456c993c4b2889f2e762 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 034b5cfccdb0456c993c4b2889f2e762, State: Initialized, Role: FOLLOWER
I20260620 11:44:04.329422 18782 tablet_bootstrap.cc:492] T 1601a6a976144e03b58a9db19004c145 P 1e4d0bc08a854ddf8b3b9e8d5dafd21f: No bootstrap required, opened a new log
I20260620 11:44:04.329427 18781 consensus_queue.cc:260] T 1601a6a976144e03b58a9db19004c145 P 034b5cfccdb0456c993c4b2889f2e762 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "034b5cfccdb0456c993c4b2889f2e762" member_type: VOTER last_known_addr { host: "127.6.234.193" port: 39713 } } peers { permanent_uuid: "1e4d0bc08a854ddf8b3b9e8d5dafd21f" member_type: VOTER last_known_addr { host: "127.6.234.195" port: 33419 } } peers { permanent_uuid: "68a7e7d7ac884d258e13f6273dee1fc9" member_type: VOTER last_known_addr { host: "127.6.234.194" port: 43959 } }
I20260620 11:44:04.329557 18782 ts_tablet_manager.cc:1397] T 1601a6a976144e03b58a9db19004c145 P 1e4d0bc08a854ddf8b3b9e8d5dafd21f: Time spent bootstrapping tablet: real 0.006s user 0.002s sys 0.000s
I20260620 11:44:04.329830 18781 ts_tablet_manager.cc:1428] T 1601a6a976144e03b58a9db19004c145 P 034b5cfccdb0456c993c4b2889f2e762: Time spent starting tablet: real 0.004s user 0.000s sys 0.003s
I20260620 11:44:04.330021 18491 heartbeater.cc:503] Master 127.6.234.254:42195 was elected leader, sending a full tablet report...
I20260620 11:44:04.331878 18782 raft_consensus.cc:348] T 1601a6a976144e03b58a9db19004c145 P 1e4d0bc08a854ddf8b3b9e8d5dafd21f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "034b5cfccdb0456c993c4b2889f2e762" member_type: VOTER last_known_addr { host: "127.6.234.193" port: 39713 } } peers { permanent_uuid: "1e4d0bc08a854ddf8b3b9e8d5dafd21f" member_type: VOTER last_known_addr { host: "127.6.234.195" port: 33419 } } peers { permanent_uuid: "68a7e7d7ac884d258e13f6273dee1fc9" member_type: VOTER last_known_addr { host: "127.6.234.194" port: 43959 } }
I20260620 11:44:04.332098 18782 raft_consensus.cc:374] T 1601a6a976144e03b58a9db19004c145 P 1e4d0bc08a854ddf8b3b9e8d5dafd21f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 11:44:04.332173 18782 raft_consensus.cc:729] T 1601a6a976144e03b58a9db19004c145 P 1e4d0bc08a854ddf8b3b9e8d5dafd21f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1e4d0bc08a854ddf8b3b9e8d5dafd21f, State: Initialized, Role: FOLLOWER
I20260620 11:44:04.332360 18782 consensus_queue.cc:260] T 1601a6a976144e03b58a9db19004c145 P 1e4d0bc08a854ddf8b3b9e8d5dafd21f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "034b5cfccdb0456c993c4b2889f2e762" member_type: VOTER last_known_addr { host: "127.6.234.193" port: 39713 } } peers { permanent_uuid: "1e4d0bc08a854ddf8b3b9e8d5dafd21f" member_type: VOTER last_known_addr { host: "127.6.234.195" port: 33419 } } peers { permanent_uuid: "68a7e7d7ac884d258e13f6273dee1fc9" member_type: VOTER last_known_addr { host: "127.6.234.194" port: 43959 } }
I20260620 11:44:04.332667 18782 ts_tablet_manager.cc:1428] T 1601a6a976144e03b58a9db19004c145 P 1e4d0bc08a854ddf8b3b9e8d5dafd21f: Time spent starting tablet: real 0.003s user 0.002s sys 0.000s
I20260620 11:44:04.332827 18763 heartbeater.cc:503] Master 127.6.234.254:42195 was elected leader, sending a full tablet report...
I20260620 11:44:04.332836 18785 tablet_bootstrap.cc:492] T 1601a6a976144e03b58a9db19004c145 P 68a7e7d7ac884d258e13f6273dee1fc9: Bootstrap starting.
I20260620 11:44:04.333884 18785 tablet_bootstrap.cc:654] T 1601a6a976144e03b58a9db19004c145 P 68a7e7d7ac884d258e13f6273dee1fc9: Neither blocks nor log segments found. Creating new log.
I20260620 11:44:04.334218 18785 log.cc:824] T 1601a6a976144e03b58a9db19004c145 P 68a7e7d7ac884d258e13f6273dee1fc9: Log is configured to *not* fsync() on all Append() calls
I20260620 11:44:04.335006 18785 tablet_bootstrap.cc:492] T 1601a6a976144e03b58a9db19004c145 P 68a7e7d7ac884d258e13f6273dee1fc9: No bootstrap required, opened a new log
I20260620 11:44:04.335068 18785 ts_tablet_manager.cc:1397] T 1601a6a976144e03b58a9db19004c145 P 68a7e7d7ac884d258e13f6273dee1fc9: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
Jun 20 11:44:04 dist-test-slave-cjc6 krb5kdc[17803](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781955843, etypes {rep=17 tkt=17 ses=17}, kudu/127.6.234.254@KRBTEST.COM for HTTP/127.6.234.212@KRBTEST.COM
I20260620 11:44:04.337100 18785 raft_consensus.cc:348] T 1601a6a976144e03b58a9db19004c145 P 68a7e7d7ac884d258e13f6273dee1fc9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "034b5cfccdb0456c993c4b2889f2e762" member_type: VOTER last_known_addr { host: "127.6.234.193" port: 39713 } } peers { permanent_uuid: "1e4d0bc08a854ddf8b3b9e8d5dafd21f" member_type: VOTER last_known_addr { host: "127.6.234.195" port: 33419 } } peers { permanent_uuid: "68a7e7d7ac884d258e13f6273dee1fc9" member_type: VOTER last_known_addr { host: "127.6.234.194" port: 43959 } }
I20260620 11:44:04.337255 18785 raft_consensus.cc:374] T 1601a6a976144e03b58a9db19004c145 P 68a7e7d7ac884d258e13f6273dee1fc9 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 11:44:04.337285 18785 raft_consensus.cc:729] T 1601a6a976144e03b58a9db19004c145 P 68a7e7d7ac884d258e13f6273dee1fc9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 68a7e7d7ac884d258e13f6273dee1fc9, State: Initialized, Role: FOLLOWER
I20260620 11:44:04.337383 18785 consensus_queue.cc:260] T 1601a6a976144e03b58a9db19004c145 P 68a7e7d7ac884d258e13f6273dee1fc9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "034b5cfccdb0456c993c4b2889f2e762" member_type: VOTER last_known_addr { host: "127.6.234.193" port: 39713 } } peers { permanent_uuid: "1e4d0bc08a854ddf8b3b9e8d5dafd21f" member_type: VOTER last_known_addr { host: "127.6.234.195" port: 33419 } } peers { permanent_uuid: "68a7e7d7ac884d258e13f6273dee1fc9" member_type: VOTER last_known_addr { host: "127.6.234.194" port: 43959 } }
I20260620 11:44:04.337601 18785 ts_tablet_manager.cc:1428] T 1601a6a976144e03b58a9db19004c145 P 68a7e7d7ac884d258e13f6273dee1fc9: Time spent starting tablet: real 0.003s user 0.002s sys 0.000s
I20260620 11:44:04.337790 18629 heartbeater.cc:503] Master 127.6.234.254:42195 was elected leader, sending a full tablet report...
W20260620 11:44:04.439738 18630 tablet.cc:2369] T 1601a6a976144e03b58a9db19004c145 P 68a7e7d7ac884d258e13f6273dee1fc9: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260620 11:44:04.472229 18492 tablet.cc:2369] T 1601a6a976144e03b58a9db19004c145 P 034b5cfccdb0456c993c4b2889f2e762: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260620 11:44:04.495390 18764 tablet.cc:2369] T 1601a6a976144e03b58a9db19004c145 P 1e4d0bc08a854ddf8b3b9e8d5dafd21f: Can't schedule compaction. Clean time has not been advanced past its initial value.
Jun 20 11:44:04 dist-test-slave-cjc6 krb5kdc[17803](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781955843, etypes {rep=17 tkt=17 ses=17}, kudu/127.6.234.254@KRBTEST.COM for HTTP/127.6.234.212@KRBTEST.COM
I20260620 11:44:04.639472 18786 raft_consensus.cc:482] T 1601a6a976144e03b58a9db19004c145 P 034b5cfccdb0456c993c4b2889f2e762 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260620 11:44:04.639632 18786 raft_consensus.cc:504] T 1601a6a976144e03b58a9db19004c145 P 034b5cfccdb0456c993c4b2889f2e762 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "034b5cfccdb0456c993c4b2889f2e762" member_type: VOTER last_known_addr { host: "127.6.234.193" port: 39713 } } peers { permanent_uuid: "1e4d0bc08a854ddf8b3b9e8d5dafd21f" member_type: VOTER last_known_addr { host: "127.6.234.195" port: 33419 } } peers { permanent_uuid: "68a7e7d7ac884d258e13f6273dee1fc9" member_type: VOTER last_known_addr { host: "127.6.234.194" port: 43959 } }
I20260620 11:44:04.640048 18786 leader_election.cc:288] T 1601a6a976144e03b58a9db19004c145 P 034b5cfccdb0456c993c4b2889f2e762 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 1e4d0bc08a854ddf8b3b9e8d5dafd21f (127.6.234.195:33419), 68a7e7d7ac884d258e13f6273dee1fc9 (127.6.234.194:43959)
I20260620 11:44:04.650772 18780 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 11:44:04.640387 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.6.234.193:46565 (local address 127.6.234.194:43959)
0620 11:44:04.640481 (+ 94us) server_negotiation.cc:206] Beginning negotiation
0620 11:44:04.640484 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0620 11:44:04.641456 (+ 972us) server_negotiation.cc:407] Connection header received
0620 11:44:04.641677 (+ 221us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 11:44:04.641678 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 11:44:04.641720 (+ 42us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 11:44:04.641761 (+ 41us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 11:44:04.643092 (+ 1331us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 11:44:04.643825 (+ 733us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 11:44:04.648999 (+ 5174us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 11:44:04.649516 (+ 517us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 11:44:04.649562 (+ 46us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 11:44:04.649571 (+ 9us) server_negotiation.cc:1035] Waiting for connection context
0620 11:44:04.650641 (+ 1070us) server_negotiation.cc:299] Negotiation successful
0620 11:44:04.650688 (+ 47us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":61}
I20260620 11:44:04.651402 18792 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 11:44:04.640191 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.234.194:43959 (local address 127.6.234.193:46565)
0620 11:44:04.641394 (+ 1203us) negotiation.cc:107] Waiting for socket to connect
0620 11:44:04.641406 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0620 11:44:04.641588 (+ 182us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 11:44:04.642779 (+ 1191us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 11:44:04.642780 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 11:44:04.642793 (+ 13us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 11:44:04.642987 (+ 194us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 11:44:04.642990 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 11:44:04.645331 (+ 2341us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 11:44:04.645333 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 11:44:04.648792 (+ 3459us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 11:44:04.648795 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 11:44:04.650493 (+ 1698us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 11:44:04.650494 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 11:44:04.650563 (+ 69us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 11:44:04.650567 (+ 4us) client_negotiation.cc:769] Sending connection context
0620 11:44:04.651247 (+ 680us) client_negotiation.cc:240] Negotiation successful
0620 11:44:04.651312 (+ 65us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1181,"mutex_wait_us":15,"thread_start_us":95,"threads_started":1}
I20260620 11:44:04.651834 18580 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1601a6a976144e03b58a9db19004c145" candidate_uuid: "034b5cfccdb0456c993c4b2889f2e762" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "68a7e7d7ac884d258e13f6273dee1fc9" is_pre_election: true
I20260620 11:44:04.652014 18580 raft_consensus.cc:2454] T 1601a6a976144e03b58a9db19004c145 P 68a7e7d7ac884d258e13f6273dee1fc9 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 034b5cfccdb0456c993c4b2889f2e762 in term 0.
I20260620 11:44:04.652254 18381 leader_election.cc:302] T 1601a6a976144e03b58a9db19004c145 P 034b5cfccdb0456c993c4b2889f2e762 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 034b5cfccdb0456c993c4b2889f2e762, 68a7e7d7ac884d258e13f6273dee1fc9; no voters:
I20260620 11:44:04.652387 18786 raft_consensus.cc:2781] T 1601a6a976144e03b58a9db19004c145 P 034b5cfccdb0456c993c4b2889f2e762 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260620 11:44:04.652424 18786 raft_consensus.cc:482] T 1601a6a976144e03b58a9db19004c145 P 034b5cfccdb0456c993c4b2889f2e762 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260620 11:44:04.652443 18786 raft_consensus.cc:3037] T 1601a6a976144e03b58a9db19004c145 P 034b5cfccdb0456c993c4b2889f2e762 [term 0 FOLLOWER]: Advancing to term 1
I20260620 11:44:04.653314 18779 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 11:44:04.640482 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.6.234.193:51321 (local address 127.6.234.195:33419)
0620 11:44:04.640524 (+ 42us) server_negotiation.cc:206] Beginning negotiation
0620 11:44:04.640526 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 11:44:04.649903 (+ 9377us) server_negotiation.cc:407] Connection header received
0620 11:44:04.650009 (+ 106us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 11:44:04.650011 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 11:44:04.650022 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 11:44:04.650070 (+ 48us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 11:44:04.650278 (+ 208us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 11:44:04.650989 (+ 711us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 11:44:04.652384 (+ 1395us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 11:44:04.652926 (+ 542us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 11:44:04.652975 (+ 49us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 11:44:04.652982 (+ 7us) server_negotiation.cc:1035] Waiting for connection context
0620 11:44:04.653194 (+ 212us) server_negotiation.cc:299] Negotiation successful
0620 11:44:04.653247 (+ 53us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":23}
I20260620 11:44:04.653297 18786 raft_consensus.cc:504] T 1601a6a976144e03b58a9db19004c145 P 034b5cfccdb0456c993c4b2889f2e762 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "034b5cfccdb0456c993c4b2889f2e762" member_type: VOTER last_known_addr { host: "127.6.234.193" port: 39713 } } peers { permanent_uuid: "1e4d0bc08a854ddf8b3b9e8d5dafd21f" member_type: VOTER last_known_addr { host: "127.6.234.195" port: 33419 } } peers { permanent_uuid: "68a7e7d7ac884d258e13f6273dee1fc9" member_type: VOTER last_known_addr { host: "127.6.234.194" port: 43959 } }
I20260620 11:44:04.653421 18786 leader_election.cc:288] T 1601a6a976144e03b58a9db19004c145 P 034b5cfccdb0456c993c4b2889f2e762 [CANDIDATE]: Term 1 election: Requested vote from peers 1e4d0bc08a854ddf8b3b9e8d5dafd21f (127.6.234.195:33419), 68a7e7d7ac884d258e13f6273dee1fc9 (127.6.234.194:43959)
I20260620 11:44:04.653564 18793 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 11:44:04.640270 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.234.195:33419 (local address 127.6.234.193:51321)
0620 11:44:04.649809 (+ 9539us) negotiation.cc:107] Waiting for socket to connect
0620 11:44:04.649821 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0620 11:44:04.649937 (+ 116us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 11:44:04.650115 (+ 178us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 11:44:04.650116 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 11:44:04.650118 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 11:44:04.650191 (+ 73us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 11:44:04.650193 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 11:44:04.651115 (+ 922us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 11:44:04.651116 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 11:44:04.652278 (+ 1162us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 11:44:04.652281 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 11:44:04.653042 (+ 761us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 11:44:04.653043 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 11:44:04.653112 (+ 69us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 11:44:04.653115 (+ 3us) client_negotiation.cc:769] Sending connection context
0620 11:44:04.653428 (+ 313us) client_negotiation.cc:240] Negotiation successful
0620 11:44:04.653469 (+ 41us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":9517,"thread_start_us":89,"threads_started":1}
I20260620 11:44:04.653856 18580 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1601a6a976144e03b58a9db19004c145" candidate_uuid: "034b5cfccdb0456c993c4b2889f2e762" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "68a7e7d7ac884d258e13f6273dee1fc9"
I20260620 11:44:04.653929 18580 raft_consensus.cc:3037] T 1601a6a976144e03b58a9db19004c145 P 68a7e7d7ac884d258e13f6273dee1fc9 [term 0 FOLLOWER]: Advancing to term 1
I20260620 11:44:04.654035 18714 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1601a6a976144e03b58a9db19004c145" candidate_uuid: "034b5cfccdb0456c993c4b2889f2e762" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1e4d0bc08a854ddf8b3b9e8d5dafd21f" is_pre_election: true
I20260620 11:44:04.654174 18714 raft_consensus.cc:2454] T 1601a6a976144e03b58a9db19004c145 P 1e4d0bc08a854ddf8b3b9e8d5dafd21f [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 034b5cfccdb0456c993c4b2889f2e762 in term 0.
I20260620 11:44:04.654295 18713 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1601a6a976144e03b58a9db19004c145" candidate_uuid: "034b5cfccdb0456c993c4b2889f2e762" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1e4d0bc08a854ddf8b3b9e8d5dafd21f"
I20260620 11:44:04.654350 18713 raft_consensus.cc:3037] T 1601a6a976144e03b58a9db19004c145 P 1e4d0bc08a854ddf8b3b9e8d5dafd21f [term 0 FOLLOWER]: Advancing to term 1
I20260620 11:44:04.654708 18580 raft_consensus.cc:2454] T 1601a6a976144e03b58a9db19004c145 P 68a7e7d7ac884d258e13f6273dee1fc9 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 034b5cfccdb0456c993c4b2889f2e762 in term 1.
I20260620 11:44:04.654968 18381 leader_election.cc:302] T 1601a6a976144e03b58a9db19004c145 P 034b5cfccdb0456c993c4b2889f2e762 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 034b5cfccdb0456c993c4b2889f2e762, 68a7e7d7ac884d258e13f6273dee1fc9; no voters:
I20260620 11:44:04.655076 18786 raft_consensus.cc:2781] T 1601a6a976144e03b58a9db19004c145 P 034b5cfccdb0456c993c4b2889f2e762 [term 1 FOLLOWER]: Leader election won for term 1
I20260620 11:44:04.655169 18713 raft_consensus.cc:2454] T 1601a6a976144e03b58a9db19004c145 P 1e4d0bc08a854ddf8b3b9e8d5dafd21f [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 034b5cfccdb0456c993c4b2889f2e762 in term 1.
I20260620 11:44:04.655218 18786 raft_consensus.cc:686] T 1601a6a976144e03b58a9db19004c145 P 034b5cfccdb0456c993c4b2889f2e762 [term 1 LEADER]: Becoming Leader. State: Replica: 034b5cfccdb0456c993c4b2889f2e762, State: Running, Role: LEADER
I20260620 11:44:04.655301 18786 consensus_queue.cc:237] T 1601a6a976144e03b58a9db19004c145 P 034b5cfccdb0456c993c4b2889f2e762 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "034b5cfccdb0456c993c4b2889f2e762" member_type: VOTER last_known_addr { host: "127.6.234.193" port: 39713 } } peers { permanent_uuid: "1e4d0bc08a854ddf8b3b9e8d5dafd21f" member_type: VOTER last_known_addr { host: "127.6.234.195" port: 33419 } } peers { permanent_uuid: "68a7e7d7ac884d258e13f6273dee1fc9" member_type: VOTER last_known_addr { host: "127.6.234.194" port: 43959 } }
I20260620 11:44:04.656090 18279 catalog_manager.cc:5495] T 1601a6a976144e03b58a9db19004c145 P 034b5cfccdb0456c993c4b2889f2e762 reported cstate change: term changed from 0 to 1, leader changed from <none> to 034b5cfccdb0456c993c4b2889f2e762 (127.6.234.193). New cstate: current_term: 1 leader_uuid: "034b5cfccdb0456c993c4b2889f2e762" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "034b5cfccdb0456c993c4b2889f2e762" member_type: VOTER last_known_addr { host: "127.6.234.193" port: 39713 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "1e4d0bc08a854ddf8b3b9e8d5dafd21f" member_type: VOTER last_known_addr { host: "127.6.234.195" port: 33419 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "68a7e7d7ac884d258e13f6273dee1fc9" member_type: VOTER last_known_addr { host: "127.6.234.194" port: 43959 } health_report { overall_health: UNKNOWN } } }
I20260620 11:44:04.744833 18778 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 11:44:04.741354 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53588 (local address 127.6.234.193:39713)
0620 11:44:04.741398 (+ 44us) server_negotiation.cc:206] Beginning negotiation
0620 11:44:04.741400 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 11:44:04.741407 (+ 7us) server_negotiation.cc:407] Connection header received
0620 11:44:04.741448 (+ 41us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 11:44:04.741448 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 11:44:04.741469 (+ 21us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 11:44:04.741548 (+ 79us) server_negotiation.cc:226] Negotiated authn=TOKEN
0620 11:44:04.741758 (+ 210us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 11:44:04.742529 (+ 771us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 11:44:04.743615 (+ 1086us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 11:44:04.743919 (+ 304us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 11:44:04.744013 (+ 94us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 11:44:04.744214 (+ 201us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0620 11:44:04.744379 (+ 165us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0620 11:44:04.744457 (+ 78us) server_negotiation.cc:1035] Waiting for connection context
0620 11:44:04.744723 (+ 266us) server_negotiation.cc:299] Negotiation successful
0620 11:44:04.744748 (+ 25us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":22}
I20260620 11:44:04.748014 18580 raft_consensus.cc:1261] T 1601a6a976144e03b58a9db19004c145 P 68a7e7d7ac884d258e13f6273dee1fc9 [term 1 FOLLOWER]: Refusing update from remote peer 034b5cfccdb0456c993c4b2889f2e762: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 11:44:04.748334 18713 raft_consensus.cc:1261] T 1601a6a976144e03b58a9db19004c145 P 1e4d0bc08a854ddf8b3b9e8d5dafd21f [term 1 FOLLOWER]: Refusing update from remote peer 034b5cfccdb0456c993c4b2889f2e762: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 11:44:04.748766 18786 consensus_queue.cc:1035] T 1601a6a976144e03b58a9db19004c145 P 034b5cfccdb0456c993c4b2889f2e762 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1e4d0bc08a854ddf8b3b9e8d5dafd21f" member_type: VOTER last_known_addr { host: "127.6.234.195" port: 33419 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 11:44:04.749187 18786 consensus_queue.cc:1035] T 1601a6a976144e03b58a9db19004c145 P 034b5cfccdb0456c993c4b2889f2e762 [LEADER]: Connected to new peer: Peer: permanent_uuid: "68a7e7d7ac884d258e13f6273dee1fc9" member_type: VOTER last_known_addr { host: "127.6.234.194" port: 43959 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 11:44:04.760082 18797 mvcc.cc:205] Tried to move back new op lower bound from 7298891140084129792 to 7298891139708469248. Current Snapshot: MvccSnapshot[applied={T|T < 7298891140084129792}]
I20260620 11:44:04.760841 18800 mvcc.cc:205] Tried to move back new op lower bound from 7298891140084129792 to 7298891139708469248. Current Snapshot: MvccSnapshot[applied={T|T < 7298891140084129792}]
I20260620 11:44:04.774081 18279 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:48042:
table { table_name: "test-table" } modify_external_catalogs: true
I20260620 11:44:04.774194 18279 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:48042:
table { table_name: "test-table" } modify_external_catalogs: true
I20260620 11:44:04.775036 18279 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 062b1d98bcfd4f7d9ed182c3416809dd: Sending DeleteTablet for 3 replicas of tablet 1601a6a976144e03b58a9db19004c145
I20260620 11:44:04.775744 18415 tablet_service.cc:1514] Processing DeleteTablet for tablet 1601a6a976144e03b58a9db19004c145 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 11:44:04 UTC) from {username='kudu', principal='kudu/127.6.234.254@KRBTEST.COM'} at 127.0.0.1:53574
I20260620 11:44:04.775951 18697 tablet_service.cc:1514] Processing DeleteTablet for tablet 1601a6a976144e03b58a9db19004c145 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 11:44:04 UTC) from {username='kudu', principal='kudu/127.6.234.254@KRBTEST.COM'} at 127.0.0.1:60312
I20260620 11:44:04.776144 18563 tablet_service.cc:1514] Processing DeleteTablet for tablet 1601a6a976144e03b58a9db19004c145 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 11:44:04 UTC) from {username='kudu', principal='kudu/127.6.234.254@KRBTEST.COM'} at 127.0.0.1:52632
I20260620 11:44:04.776747 7083 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task1JRg7N/build/release/bin/kudu with pid 18360
I20260620 11:44:04.781214 18807 tablet_replica.cc:330] T 1601a6a976144e03b58a9db19004c145 P 1e4d0bc08a854ddf8b3b9e8d5dafd21f: stopping tablet replica
I20260620 11:44:04.781374 18807 raft_consensus.cc:2229] T 1601a6a976144e03b58a9db19004c145 P 1e4d0bc08a854ddf8b3b9e8d5dafd21f [term 1 FOLLOWER]: Raft consensus shutting down.
I20260620 11:44:04.781101 18808 tablet_replica.cc:330] T 1601a6a976144e03b58a9db19004c145 P 68a7e7d7ac884d258e13f6273dee1fc9: stopping tablet replica
I20260620 11:44:04.781749 18808 raft_consensus.cc:2229] T 1601a6a976144e03b58a9db19004c145 P 68a7e7d7ac884d258e13f6273dee1fc9 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260620 11:44:04.781944 18807 raft_consensus.cc:2258] T 1601a6a976144e03b58a9db19004c145 P 1e4d0bc08a854ddf8b3b9e8d5dafd21f [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 11:44:04.782135 18808 raft_consensus.cc:2258] T 1601a6a976144e03b58a9db19004c145 P 68a7e7d7ac884d258e13f6273dee1fc9 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 11:44:04.782768 18807 ts_tablet_manager.cc:1905] T 1601a6a976144e03b58a9db19004c145 P 1e4d0bc08a854ddf8b3b9e8d5dafd21f: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 11:44:04.787627 18807 ts_tablet_manager.cc:1918] T 1601a6a976144e03b58a9db19004c145 P 1e4d0bc08a854ddf8b3b9e8d5dafd21f: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260620 11:44:04.787772 18807 log.cc:1196] T 1601a6a976144e03b58a9db19004c145 P 1e4d0bc08a854ddf8b3b9e8d5dafd21f: Deleting WAL directory at /tmp/dist-test-task1JRg7N/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781955746769051-7083-0/minicluster-data/ts-2/wal/wals/1601a6a976144e03b58a9db19004c145
I20260620 11:44:04.788108 18807 ts_tablet_manager.cc:1939] T 1601a6a976144e03b58a9db19004c145 P 1e4d0bc08a854ddf8b3b9e8d5dafd21f: Deleting consensus metadata
I20260620 11:44:04.788604 18263 catalog_manager.cc:4841] TS 1e4d0bc08a854ddf8b3b9e8d5dafd21f (127.6.234.195:33419): tablet 1601a6a976144e03b58a9db19004c145 (table test-table [id=7eeadee59b704a818724666f04ea2d0a]) successfully deleted
W20260620 11:44:04.789150 18264 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.6.234.193:39713 (error 108)
I20260620 11:44:04.789515 18808 ts_tablet_manager.cc:1905] T 1601a6a976144e03b58a9db19004c145 P 68a7e7d7ac884d258e13f6273dee1fc9: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 11:44:04.789978 18775 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 11:44:04.789863 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.234.193:39713 (local address 127.0.0.1:53596)
0620 11:44:04.789908 (+ 45us) negotiation.cc:107] Waiting for socket to connect
0620 11:44:04.789945 (+ 37us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.234.193:39713: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":26}
W20260620 11:44:04.790161 18264 catalog_manager.cc:4584] TS 034b5cfccdb0456c993c4b2889f2e762 (127.6.234.193:39713): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 1601a6a976144e03b58a9db19004c145: Network error: Client connection negotiation failed: client connection to 127.6.234.193:39713: connect: Connection refused (error 111)
I20260620 11:44:04.790390 7083 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task1JRg7N/build/release/bin/kudu with pid 18499
I20260620 11:44:04.796342 18775 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 11:44:04.796228 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.234.194:43959 (local address 127.0.0.1:52646)
0620 11:44:04.796279 (+ 51us) negotiation.cc:107] Waiting for socket to connect
0620 11:44:04.796306 (+ 27us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.234.194:43959: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":32}
I20260620 11:44:04.796366 7083 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task1JRg7N/build/release/bin/kudu with pid 18634
I20260620 11:44:04.803524 7083 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task1JRg7N/build/release/bin/kudu with pid 18247
I20260620 11:44:04.809161 7083 mini_ranger_kms.cc:60] Stopping Ranger KMS...
I20260620 11:44:05.153285 7083 mini_ranger_kms.cc:62] Stopped Ranger KMS
I20260620 11:44:05.153447 7083 mini_ranger.cc:66] Stopping Ranger...
I20260620 11:44:05.497903 7083 mini_ranger.cc:68] Stopped Ranger
2026-06-20 11:44:05.498 UTC [17833] LOG: received smart shutdown request
2026-06-20 11:44:05.501 UTC [17833] LOG: background worker "logical replication launcher" (PID 17842) exited with exit code 1
2026-06-20 11:44:05.502 UTC [17837] LOG: shutting down
2026-06-20 11:44:05.666 UTC [17833] LOG: database system is shut down
2026-06-20T11:44:05Z chronyd exiting
[ OK ] SecurityITest.TestEncryptionWithKMSIntegration (45765 ms)
[ RUN ] SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers
Loading random data
Initializing database '/tmp/dist-test-task1JRg7N/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781955746769051-7083-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 20 11:44:05 dist-test-slave-cjc6 krb5kdc[18818](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 20 11:44:05 dist-test-slave-cjc6 krb5kdc[18818](info): set up 2 sockets
Jun 20 11:44:05 dist-test-slave-cjc6 krb5kdc[18818](info): commencing operation
krb5kdc: starting...
W20260620 11:44:06.805835 7083 mini_kdc.cc:121] Time spent starting KDC: real 1.015s user 0.000s sys 0.006s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task1JRg7N/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781955746769051-7083-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task1JRg7N/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781955746769051-7083-0/krb5kdc/test-user.keytab.
Jun 20 11:44:06 dist-test-slave-cjc6 krb5kdc[18818](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781955846, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-20T11:44:06Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-20T11:44:06Z Disabled control of system clock
WARNING: no policy specified for rangeradmin/127.6.234.212@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "rangeradmin/127.6.234.212@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal rangeradmin/127.6.234.212@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task1JRg7N/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781955746769051-7083-0/krb5kdc/rangeradmin_127.6.234.212@KRBTEST.COM.keytab.
Entry for principal rangeradmin/127.6.234.212@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task1JRg7N/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781955746769051-7083-0/krb5kdc/rangeradmin_127.6.234.212@KRBTEST.COM.keytab.
WARNING: no policy specified for rangerlookup/127.6.234.212@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "rangerlookup/127.6.234.212@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal rangerlookup/127.6.234.212@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task1JRg7N/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781955746769051-7083-0/krb5kdc/rangerlookup_127.6.234.212@KRBTEST.COM.keytab.
Entry for principal rangerlookup/127.6.234.212@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task1JRg7N/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781955746769051-7083-0/krb5kdc/rangerlookup_127.6.234.212@KRBTEST.COM.keytab.
WARNING: no policy specified for HTTP/127.6.234.212@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "HTTP/127.6.234.212@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.6.234.212@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task1JRg7N/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781955746769051-7083-0/krb5kdc/HTTP_127.6.234.212@KRBTEST.COM.keytab.
Entry for principal HTTP/127.6.234.212@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task1JRg7N/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781955746769051-7083-0/krb5kdc/HTTP_127.6.234.212@KRBTEST.COM.keytab.
I20260620 11:44:06.968324 7083 mini_postgres.cc:62] Running initdb...
The files belonging to this database system will be owned by user "slave".
This user must also own the server process.
The database cluster will be initialized with locale "C".
The default database encoding has accordingly been set to "SQL_ASCII".
The default text search configuration will be set to "english".
Data page checksums are disabled.
creating directory /tmp/dist-test-task1JRg7N/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781955746769051-7083-0/minicluster-data/postgres ... ok
creating subdirectories ... ok
selecting dynamic shared memory implementation ... posix
selecting default max_connections ... 100
selecting default shared_buffers ... 128MB
selecting default time zone ... Etc/UTC
creating configuration files ... ok
running bootstrap script ... ok
performing post-bootstrap initialization ... ok
syncing data to disk ... ok
initdb: warning: enabling "trust" authentication for local connections
You can change this by editing pg_hba.conf or using the option -A, or
--auth-local and --auth-host, the next time you run initdb.
Success. You can now start the database server using:
/tmp/dist-test-task1JRg7N/build/release/bin/postgres/pg_ctl -D /tmp/dist-test-task1JRg7N/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781955746769051-7083-0/minicluster-data/postgres -l logfile start
2026-06-20 11:44:09.424 UTC [18848] LOG: starting PostgreSQL 12.2 on x86_64-pc-linux-gnu, compiled by cc (GCC) 8.3.1 20190311 (Red Hat 8.3.1-3), 64-bit
2026-06-20 11:44:09.424 UTC [18848] LOG: listening on IPv4 address "127.6.234.212", port 46445
2026-06-20 11:44:09.427 UTC [18848] LOG: listening on Unix socket "/tmp/.s.PGSQL.46445"
2026-06-20 11:44:09.448 UTC [18851] LOG: database system was shut down at 2026-06-20 11:44:08 UTC
2026-06-20 11:44:09.451 UTC [18848] LOG: database system is ready to accept connections
I20260620 11:44:10.425503 7083 mini_postgres.cc:96] Postgres bound to 46445
2026-06-20 11:44:10.431 UTC [18859] FATAL: database "slave" does not exist
127.6.234.212:46445 - accepting connections
I20260620 11:44:10.432080 7083 mini_ranger.cc:161] Starting Ranger...
I20260620 11:44:10.446343 7083 mini_ranger.cc:84] Created miniranger Postgres user
I20260620 11:44:10.916644 7083 mini_ranger.cc:87] Created ranger Postgres database
I20260620 11:44:10.916731 7083 mini_ranger.cc:178] Starting Ranger out of /tmp/dist-test-task1JRg7N/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781955746769051-7083-0/minicluster-data/ranger-admin
2026-06-20 11:44:11,200 [I] DB FLAVOR :POSTGRES
2026-06-20 11:44:11,200 [I] --------- Verifying Ranger DB connection ---------
2026-06-20 11:44:11,200 [I] Checking connection..
2026-06-20 11:44:11,201 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task1JRg7N/build/release/bin/postgresql.jar:/tmp/dist-test-task1JRg7N/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.6.234.212:46445/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select 1;"
2026-06-20 11:44:11,433 [I] Checking connection passed.
2026-06-20 11:44:11,433 [I] --------- Verifying version history table ---------
2026-06-20 11:44:11,434 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task1JRg7N/build/release/bin/postgresql.jar:/tmp/dist-test-task1JRg7N/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.6.234.212:46445/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_db_version_h') as temp;"
2026-06-20 11:44:11,684 [I] Table x_db_version_h does not exist in database ranger
2026-06-20 11:44:11,685 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task1JRg7N/build/release/bin/postgresql.jar:/tmp/dist-test-task1JRg7N/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.6.234.212:46445/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_db_version_h') as temp;"
2026-06-20 11:44:11,947 [I] Table x_db_version_h does not exist in database ranger
2026-06-20 11:44:11,948 [I] Importing x_db_version_h table schema to database ranger from file: create_dbversion_catalog.sql
2026-06-20 11:44:11,948 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task1JRg7N/build/release/bin/postgresql.jar:/tmp/dist-test-task1JRg7N/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.6.234.212:46445/ranger -u miniranger -p '********' -noheader -trim -c \; -input /tmp/dist-test-task1JRg7N/thirdparty/src/ranger-2.1.0-admin/db/postgres/create_dbversion_catalog.sql
2026-06-20 11:44:12.180 UTC [18923] WARNING: there is no transaction in progress
2026-06-20 11:44:12,186 [I] create_dbversion_catalog.sql file imported successfully
2026-06-20 11:44:12,187 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task1JRg7N/build/release/bin/postgresql.jar:/tmp/dist-test-task1JRg7N/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.6.234.212:46445/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_db_version_h') as temp;"
2026-06-20 11:44:12,415 [I] Table x_db_version_h already exists in database 'ranger'
2026-06-20 11:44:12,416 [I] --------- Importing Ranger Core DB Schema ---------
2026-06-20 11:44:12,416 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task1JRg7N/build/release/bin/postgresql.jar:/tmp/dist-test-task1JRg7N/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.6.234.212:46445/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'CORE_DB_SCHEMA' and active = 'Y';"
2026-06-20 11:44:12,662 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task1JRg7N/build/release/bin/postgresql.jar:/tmp/dist-test-task1JRg7N/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.6.234.212:46445/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'CORE_DB_SCHEMA' and active = 'N';"
2026-06-20 11:44:12,886 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task1JRg7N/build/release/bin/postgresql.jar:/tmp/dist-test-task1JRg7N/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.6.234.212:46445/ranger -u miniranger -p '********' -noheader -trim -c \; -query "insert into x_db_version_h (version, inst_at, inst_by, updated_at, updated_by,active) values ('CORE_DB_SCHEMA', current_timestamp, 'Ranger 2.1.0', current_timestamp, 'dist-test-slave-cjc6.c.gcp-upstream.internal','N') ;"
2026-06-20 11:44:13,114 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task1JRg7N/build/release/bin/postgresql.jar:/tmp/dist-test-task1JRg7N/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.6.234.212:46445/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_portal_user') as temp;"
2026-06-20 11:44:13,375 [I] Table x_portal_user does not exist in database ranger
2026-06-20 11:44:13,375 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task1JRg7N/build/release/bin/postgresql.jar:/tmp/dist-test-task1JRg7N/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.6.234.212:46445/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_policy_ref_group') as temp;"
2026-06-20 11:44:13,619 [I] Table x_policy_ref_group does not exist in database ranger
2026-06-20 11:44:13,619 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task1JRg7N/build/release/bin/postgresql.jar:/tmp/dist-test-task1JRg7N/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.6.234.212:46445/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'DB_PATCHES' and active = 'Y';"
2026-06-20 11:44:13,844 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task1JRg7N/build/release/bin/postgresql.jar:/tmp/dist-test-task1JRg7N/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.6.234.212:46445/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'JAVA_PATCHES' and active = 'Y';"
2026-06-20 11:44:14,074 [I] Importing DB schema to database ranger from file: ranger_core_db_postgres.sql
2026-06-20 11:44:14,074 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task1JRg7N/build/release/bin/postgresql.jar:/tmp/dist-test-task1JRg7N/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.6.234.212:46445/ranger -u miniranger -p '********' -noheader -trim -c \; -input /tmp/dist-test-task1JRg7N/thirdparty/src/ranger-2.1.0-admin/db/postgres/optimized/current/ranger_core_db_postgres.sql
2026-06-20 11:44:15.044 UTC [19031] WARNING: there is no transaction in progress
2026-06-20 11:44:15.060 UTC [19031] WARNING: there is no transaction in progress
2026-06-20 11:44:15.080 UTC [19031] WARNING: there is no transaction in progress
2026-06-20 11:44:15.095 UTC [19031] WARNING: there is no transaction in progress
2026-06-20 11:44:15.112 UTC [19031] WARNING: there is no transaction in progress
2026-06-20 11:44:15.129 UTC [19031] WARNING: there is no transaction in progress
2026-06-20 11:44:15.183 UTC [19031] WARNING: there is no transaction in progress
2026-06-20 11:44:15.189 UTC [19031] WARNING: there is no transaction in progress
2026-06-20 11:44:15.203 UTC [19031] WARNING: there is no transaction in progress
2026-06-20 11:44:15.213 UTC [19031] WARNING: there is no transaction in progress
2026-06-20 11:44:15.223 UTC [19031] WARNING: there is no transaction in progress
2026-06-20 11:44:15.236 UTC [19031] WARNING: there is no transaction in progress
2026-06-20 11:44:15.246 UTC [19031] WARNING: there is no transaction in progress
2026-06-20 11:44:15.252 UTC [19031] WARNING: there is no transaction in progress
2026-06-20 11:44:15.258 UTC [19031] WARNING: there is no transaction in progress
2026-06-20 11:44:15.264 UTC [19031] WARNING: there is no transaction in progress
2026-06-20 11:44:15,884 [I] ranger_core_db_postgres.sql file imported successfully
2026-06-20 11:44:15,884 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task1JRg7N/build/release/bin/postgresql.jar:/tmp/dist-test-task1JRg7N/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.6.234.212:46445/ranger -u miniranger -p '********' -noheader -trim -c \; -query "update x_db_version_h set inst_by='Ranger 2.1.0' where active='Y' and updated_by='localhost';"
2026-06-20 11:44:16,117 [I] Patches status entries updated from base ranger version to current installed ranger version:Ranger 2.1.0
2026-06-20 11:44:16,117 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task1JRg7N/build/release/bin/postgresql.jar:/tmp/dist-test-task1JRg7N/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.6.234.212:46445/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_portal_user') as temp;"
2026-06-20 11:44:16,343 [I] Table x_portal_user already exists in database 'ranger'
2026-06-20 11:44:16,344 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task1JRg7N/build/release/bin/postgresql.jar:/tmp/dist-test-task1JRg7N/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.6.234.212:46445/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_policy_ref_group') as temp;"
2026-06-20 11:44:16,582 [I] Table x_policy_ref_group already exists in database 'ranger'
2026-06-20 11:44:16,582 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task1JRg7N/build/release/bin/postgresql.jar:/tmp/dist-test-task1JRg7N/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.6.234.212:46445/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'DB_PATCHES' and active = 'Y';"
2026-06-20 11:44:16,833 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task1JRg7N/build/release/bin/postgresql.jar:/tmp/dist-test-task1JRg7N/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.6.234.212:46445/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'JAVA_PATCHES' and active = 'Y';"
2026-06-20 11:44:17,094 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task1JRg7N/build/release/bin/postgresql.jar:/tmp/dist-test-task1JRg7N/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.6.234.212:46445/ranger -u miniranger -p '********' -noheader -trim -c \; -query "update x_db_version_h set active='Y' where version='CORE_DB_SCHEMA' and active='N' and updated_by='dist-test-slave-cjc6.c.gcp-upstream.internal';"
2026-06-20 11:44:17,321 [I] CORE_DB_SCHEMA import status has been updated
2026-06-20 11:44:17,322 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task1JRg7N/build/release/bin/postgresql.jar:/tmp/dist-test-task1JRg7N/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.6.234.212:46445/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'DB_PATCHES' and inst_by = 'Ranger 2.1.0' and active = 'Y';"
2026-06-20 11:44:17,576 [I] DB_PATCHES have already been applied
I20260620 11:44:17.581573 7083 mini_ranger.cc:191] Using Ranger class path: /tmp/dist-test-task1JRg7N/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781955746769051-7083-0/minicluster-data/ranger-admin:/tmp/dist-test-task1JRg7N/thirdparty/src/ranger-2.1.0-admin/ews/lib/*:/usr/lib/jvm/java-1.8.0-openjdk-amd64/lib/*:/tmp/dist-test-task1JRg7N/thirdparty/src/hadoop-3.2.0/*:/tmp/dist-test-task1JRg7N/build/release/bin/postgresql.jar:/tmp/dist-test-task1JRg7N/thirdparty/src/ranger-2.1.0-admin/ews/webapp
I20260620 11:44:17.581668 7083 mini_ranger.cc:193] Using host: 127.6.234.212
I20260620 11:44:17.584700 7083 mini_ranger.cc:227] Ranger admin URL: http://127.6.234.212:36853
Jun 20, 2026 11:44:17 AM org.apache.ranger.server.tomcat.EmbeddedServer getKeyManagers
WARNING: Config 'ranger.keystore.file' or 'ranger.service.https.attrib.keystore.file' is not found or contains blank value
Jun 20, 2026 11:44:17 AM org.apache.ranger.server.tomcat.EmbeddedServer getTrustManagers
WARNING: Config 'ranger.truststore.file' is not found or contains blank value!
Jun 20, 2026 11:44:17 AM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Deriving webapp folder from catalina.base property. folder=/tmp/dist-test-task1JRg7N/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781955746769051-7083-0/minicluster-data/ranger-admin/ews/webapp
Jun 20, 2026 11:44:17 AM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Webapp file =/tmp/dist-test-task1JRg7N/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781955746769051-7083-0/minicluster-data/ranger-admin/ews/webapp, webAppName = /
Jun 20, 2026 11:44:17 AM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Adding webapp [/] = path [/tmp/dist-test-task1JRg7N/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781955746769051-7083-0/minicluster-data/ranger-admin/ews/webapp] .....
Jun 20, 2026 11:44:18 AM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Finished init of webapp [/] = path [/tmp/dist-test-task1JRg7N/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781955746769051-7083-0/minicluster-data/ranger-admin/ews/webapp].
Jun 20, 2026 11:44:18 AM org.apache.ranger.server.tomcat.EmbeddedServer startServer
INFO: Server Name : miniranger
I20260620 11:44:18.691270 7083 mini_ranger.cc:160] Time spent starting Ranger: real 8.259s user 0.003s sys 0.005s
Jun 20, 2026 11:44:37 AM com.sun.jersey.api.core.PackagesResourceConfig init
INFO: Scanning for root resource and provider classes in the packages:
org.apache.ranger.rest
org.apache.ranger.common
xa.rest
Jun 20, 2026 11:44:37 AM com.sun.jersey.api.core.ScanningResourceConfig logClasses
INFO: Root resource classes found:
class org.apache.ranger.rest.XAuditREST
class org.apache.ranger.rest.SecurityZoneREST
class org.apache.ranger.rest.PublicAPIsv2
class org.apache.ranger.rest.XKeyREST
class org.apache.ranger.rest.RoleREST
class org.apache.ranger.rest.ServiceREST
class org.apache.ranger.rest.AssetREST
class org.apache.ranger.rest.XUserREST
class org.apache.ranger.rest.TagREST
class org.apache.ranger.rest.MetricsREST
class org.apache.ranger.rest.PublicAPIs
class org.apache.ranger.rest.UserREST
Jun 20, 2026 11:44:37 AM com.sun.jersey.api.core.ScanningResourceConfig logClasses
INFO: Provider classes found:
class org.apache.ranger.common.RangerJAXBContextResolver
Jun 20, 2026 11:44:38 AM com.sun.jersey.spi.spring.container.servlet.SpringServlet getContext
INFO: Using default applicationContext
Jun 20, 2026 11:44:38 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, assetREST, of type org.apache.ranger.rest.AssetREST as a root resource class
Jun 20, 2026 11:44:38 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, metricsREST, of type org.apache.ranger.rest.MetricsREST as a root resource class
Jun 20, 2026 11:44:38 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, publicAPIs, of type org.apache.ranger.rest.PublicAPIs as a root resource class
Jun 20, 2026 11:44:38 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, publicAPIsv2, of type org.apache.ranger.rest.PublicAPIsv2 as a root resource class
Jun 20, 2026 11:44:38 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, roleREST, of type org.apache.ranger.rest.RoleREST as a root resource class
Jun 20, 2026 11:44:38 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, securityZoneREST, of type org.apache.ranger.rest.SecurityZoneREST as a root resource class
Jun 20, 2026 11:44:38 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, serviceREST, of type org.apache.ranger.rest.ServiceREST as a root resource class
Jun 20, 2026 11:44:38 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, tagREST, of type org.apache.ranger.rest.TagREST as a root resource class
Jun 20, 2026 11:44:38 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, userREST, of type org.apache.ranger.rest.UserREST as a root resource class
Jun 20, 2026 11:44:38 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, XAuditREST, of type org.apache.ranger.rest.XAuditREST as a root resource class
Jun 20, 2026 11:44:38 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, XKeyREST, of type org.apache.ranger.rest.XKeyREST as a root resource class
Jun 20, 2026 11:44:38 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, XUserREST, of type org.apache.ranger.rest.XUserREST as a root resource class
Jun 20, 2026 11:44:38 AM com.sun.jersey.server.impl.application.WebApplicationImpl _initiate
INFO: Initiating Jersey application, version 'Jersey: 1.19.3 10/24/2016 03:58 PM'
Jun 20, 2026 11:44:38 AM com.sun.jersey.spi.inject.Errors processErrorMessages
WARNING: The following warnings have been detected with resource and/or provider classes:
WARNING: A HTTP GET method, public void org.apache.ranger.rest.ServiceREST.getPoliciesInExcel(javax.servlet.http.HttpServletRequest,javax.servlet.http.HttpServletResponse), MUST return a non-void type.
WARNING: A HTTP GET method, public void org.apache.ranger.rest.ServiceREST.getPoliciesInJson(javax.servlet.http.HttpServletRequest,javax.servlet.http.HttpServletResponse,java.lang.Boolean), MUST return a non-void type.
WARNING: A HTTP GET method, public void org.apache.ranger.rest.ServiceREST.getPoliciesInCsv(javax.servlet.http.HttpServletRequest,javax.servlet.http.HttpServletResponse) throws java.io.IOException, MUST return a non-void type.
I20260620 11:44:39.424175 7083 mini_ranger.cc:261] Created Kudu service
WARNING: no policy specified for rangerkms/127.6.234.212@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "rangerkms/127.6.234.212@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal rangerkms/127.6.234.212@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task1JRg7N/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781955746769051-7083-0/krb5kdc/rangerkms_127.6.234.212@KRBTEST.COM.keytab.
Entry for principal rangerkms/127.6.234.212@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task1JRg7N/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781955746769051-7083-0/krb5kdc/rangerkms_127.6.234.212@KRBTEST.COM.keytab.
WARNING: no policy specified for HTTP/127.6.234.212@KRBTEST.COM; defaulting to no policy
add_principal: Principal or policy already exists while creating "HTTP/127.6.234.212@KRBTEST.COM".
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.6.234.212@KRBTEST.COM with kvno 3, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task1JRg7N/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781955746769051-7083-0/krb5kdc/HTTP_127.6.234.212@KRBTEST.COM.keytab.
Entry for principal HTTP/127.6.234.212@KRBTEST.COM with kvno 3, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task1JRg7N/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781955746769051-7083-0/krb5kdc/HTTP_127.6.234.212@KRBTEST.COM.keytab.
WARNING: no policy specified for keyadmin@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "keyadmin@KRBTEST.COM" created.
Jun 20 11:44:39 dist-test-slave-cjc6 krb5kdc[18818](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781955879, etypes {rep=17 tkt=17 ses=17}, keyadmin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for keyadmin@KRBTEST.COM:
I20260620 11:44:39.500331 7083 mini_ranger_kms.cc:208] Starting Ranger KMS...
I20260620 11:44:39.508903 7083 mini_ranger_kms.cc:78] Created minirangerkms Postgres user
I20260620 11:44:40.339323 7083 mini_ranger_kms.cc:81] Created rangerkms Postgres database
I20260620 11:44:40.339453 7083 mini_ranger_kms.cc:226] Starting Ranger KMS out of /tmp/dist-test-task1JRg7N/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781955746769051-7083-0/minicluster-data/ranger-kms
I20260620 11:44:40.339478 7083 mini_ranger_kms.cc:227] Using postgres at 127.6.234.212:46445
2026-06-20 11:44:40,503 [I] DB FLAVOR :POSTGRES
2026-06-20 11:44:40,504 [I] --------- Verifying Ranger DB connection ---------
2026-06-20 11:44:40,504 [I] Checking connection
2026-06-20 11:44:40,504 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task1JRg7N/build/release/bin/postgresql.jar:/tmp/dist-test-task1JRg7N/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781955746769051-7083-0/minicluster-data/ranger-kms/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.6.234.212:46445/rangerkms -u rangerkms -p '********' -noheader -trim -c \; -query "SELECT 1;"
2026-06-20 11:44:40,765 [I] connection success
2026-06-20 11:44:40,765 [I] --------- Verifying Ranger DB tables ---------
2026-06-20 11:44:40,765 [I] Verifying table ranger_masterkey in database rangerkms
2026-06-20 11:44:40,765 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task1JRg7N/build/release/bin/postgresql.jar:/tmp/dist-test-task1JRg7N/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781955746769051-7083-0/minicluster-data/ranger-kms/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.6.234.212:46445/rangerkms -u rangerkms -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='rangerkms' and table_name = 'ranger_masterkey') as temp;"
2026-06-20 11:44:41,002 [I] Table ranger_masterkey does not exist in database rangerkms
2026-06-20 11:44:41,002 [I] --------- Importing Ranger Core DB Schema ---------
2026-06-20 11:44:41,002 [I] Importing db schema to database rangerkms from file: kms_core_db_postgres.sql
2026-06-20 11:44:41,002 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task1JRg7N/build/release/bin/postgresql.jar:/tmp/dist-test-task1JRg7N/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781955746769051-7083-0/minicluster-data/ranger-kms/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.6.234.212:46445/rangerkms -u rangerkms -p '********' -noheader -trim -c \; -input /tmp/dist-test-task1JRg7N/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781955746769051-7083-0/minicluster-data/ranger-kms/db/postgres/kms_core_db_postgres.sql
2026-06-20 11:44:41,274 [I] kms_core_db_postgres.sql DB schema imported successfully
I20260620 11:44:41.738063 7083 mini_ranger_kms.cc:310] Created kms service
I20260620 11:44:41.934513 7083 mini_ranger_kms.cc:326] Created kudu user
I20260620 11:44:41.976469 7083 mini_ranger_kms.cc:343] Created rangerkms user
Jun 20 11:44:41 dist-test-slave-cjc6 krb5kdc[18818](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781955879, etypes {rep=17 tkt=17 ses=17}, keyadmin@KRBTEST.COM for HTTP/127.6.234.212@KRBTEST.COM
I20260620 11:44:42.372673 7083 mini_ranger_kms.cc:384] Added ranger policy
I20260620 11:44:42.372913 7083 mini_ranger_kms.cc:238] Using RangerKMS classpath: /tmp/dist-test-task1JRg7N/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781955746769051-7083-0/minicluster-data/ranger-kms:/tmp/dist-test-task1JRg7N/build/release/bin/postgresql.jar:/tmp/dist-test-task1JRg7N/thirdparty/src/ranger-2.1.0-kms/ews/webapp/WEB-INF/classes/lib/*:/tmp/dist-test-task1JRg7N/thirdparty/src/ranger-2.1.0-kms/ews/webapp/lib/*:/usr/lib/jvm/java-1.8.0-openjdk-amd64/lib/*
I20260620 11:44:42.372941 7083 mini_ranger_kms.cc:240] Using host: 127.6.234.212
I20260620 11:44:42.378129 7083 mini_ranger_kms.cc:276] Ranger KMS PID: 19211
I20260620 11:44:42.378209 7083 mini_ranger_kms.cc:277] Ranger KMS URL: http://127.6.234.212:44851
Jun 20, 2026 11:44:43 AM org.apache.ranger.server.tomcat.EmbeddedServer getKeyManagers
WARNING: Config 'ranger.keystore.file' or 'ranger.service.https.attrib.keystore.file' is not found or contains blank value
Jun 20, 2026 11:44:43 AM org.apache.ranger.server.tomcat.EmbeddedServer getTrustManagers
WARNING: Config 'ranger.truststore.file' is not found or contains blank value!
Jun 20, 2026 11:44:43 AM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Webapp file =/tmp/dist-test-task1JRg7N/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781955746769051-7083-0/minicluster-data/ranger-kms/ews/webapp, webAppName = /kms
Jun 20, 2026 11:44:43 AM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Adding webapp [/kms] = path [/tmp/dist-test-task1JRg7N/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781955746769051-7083-0/minicluster-data/ranger-kms/ews/webapp] .....
Jun 20, 2026 11:44:43 AM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Finished init of webapp [/kms] = path [/tmp/dist-test-task1JRg7N/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781955746769051-7083-0/minicluster-data/ranger-kms/ews/webapp].
Jun 20, 2026 11:44:43 AM org.apache.ranger.server.tomcat.EmbeddedServer startServer
INFO: Server Name : minirangerkms
Jun 20 11:44:44 dist-test-slave-cjc6 krb5kdc[18818](info): AS_REQ (1 etypes {17}) 127.0.0.1: ISSUE: authtime 1781955884, etypes {rep=17 tkt=17 ses=17}, rangerkms/127.6.234.212@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 11:44:44.624701 7083 mini_ranger_kms.cc:207] Time spent starting Ranger KMS: real 5.124s user 0.006s sys 0.141s
I20260620 11:44:44.624809 7083 mini_ranger_kms.cc:397] {"name":"kuduclusterkey","cipher":"AES/CTR/NoPadding","length":128,"description":"kuduclusterkey"}
I20260620 11:44:44.624853 7083 mini_ranger_kms.cc:401] 127.6.234.212:44851/kms/v1/keys
Jun 20 11:44:45 dist-test-slave-cjc6 krb5kdc[18818](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781955884, etypes {rep=17 tkt=17 ses=17}, rangerkms/127.6.234.212@KRBTEST.COM for HTTP/127.6.234.212@KRBTEST.COM
Jun 20 11:44:45 dist-test-slave-cjc6 krb5kdc[18818](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781955884, etypes {rep=17 tkt=17 ses=17}, rangerkms/127.6.234.212@KRBTEST.COM for HTTP/127.6.234.212@KRBTEST.COM
Jun 20 11:44:47 dist-test-slave-cjc6 krb5kdc[18818](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781955887, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
WARNING: no policy specified for kudu/127.6.234.254@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.6.234.254@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.6.234.254 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task1JRg7N/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781955746769051-7083-0/krb5kdc/kudu_127.6.234.254.keytab.
Entry for principal kudu/127.6.234.254 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task1JRg7N/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781955746769051-7083-0/krb5kdc/kudu_127.6.234.254.keytab.
I20260620 11:44:47.574640 7083 external_mini_cluster.cc:1326] Running /tmp/dist-test-task1JRg7N/build/release/bin/kudu
/tmp/dist-test-task1JRg7N/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task1JRg7N/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781955746769051-7083-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task1JRg7N/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781955746769051-7083-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task1JRg7N/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781955746769051-7083-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task1JRg7N/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781955746769051-7083-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.6.234.254:36781
--webserver_interface=127.6.234.254
--webserver_port=0
--keytab_file=/tmp/dist-test-task1JRg7N/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781955746769051-7083-0/krb5kdc/kudu_127.6.234.254.keytab
--principal=kudu/127.6.234.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.6.234.212:38181
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.6.234.254:36781
--ranger_config_path=/tmp/dist-test-task1JRg7N/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781955746769051-7083-0/minicluster-data/ranger-client
--trusted_user_acl=test-admin
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.6.234.212:44851/kms
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task1JRg7N/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781955746769051-7083-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task1JRg7N/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781955746769051-7083-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task1JRg7N/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781955746769051-7083-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 11:44:47.659293 19262 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 11:44:47.659552 19262 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 11:44:47.659590 19262 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 11:44:47.660724 19262 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 11:44:47.660805 19262 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 11:44:47.660830 19262 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 11:44:47.660851 19262 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 11:44:47.660880 19262 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 11:44:47.662520 19262 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.6.234.212:38181
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-task1JRg7N/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781955746769051-7083-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task1JRg7N/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781955746769051-7083-0/minicluster-data/master-0/wal
--ranger_kms_url=127.6.234.212:44851/kms
--trusted_user_acl=<redacted>
--ipki_ca_key_size=768
--master_addresses=127.6.234.254:36781
--ranger_config_path=/tmp/dist-test-task1JRg7N/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781955746769051-7083-0/minicluster-data/ranger-client
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task1JRg7N/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781955746769051-7083-0/krb5kdc/kudu_127.6.234.254.keytab
--principal=kudu/127.6.234.254
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.6.234.254:36781
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task1JRg7N/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781955746769051-7083-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.6.234.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.19262
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task1JRg7N/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781955746769051-7083-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 20 Jun 2026 10:32:37 UTC on ccycloud.quasar-bgyofv.root.comops.site
build id 4457173
I20260620 11:44:47.662801 19262 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 11:44:47.663033 19262 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 11:44:47.667958 19270 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 11:44:47.667970 19268 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 11:44:47.667970 19267 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 11:44:47.668255 19262 server_base.cc:1034] running on GCE node
I20260620 11:44:47.668496 19262 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 11:44:47.668782 19262 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 11:44:47.669943 19262 hybrid_clock.cc:648] HybridClock initialized: now 1781955887669941 us; error 41 us; skew 500 ppm
Jun 20 11:44:47 dist-test-slave-cjc6 krb5kdc[18818](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781955887, etypes {rep=17 tkt=17 ses=17}, kudu/127.6.234.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 11:44:47.676864 19262 init.cc:377] Logged in from keytab as kudu/127.6.234.254@KRBTEST.COM (short username kudu)
I20260620 11:44:47.677364 19262 webserver.cc:460] Webserver started at http://127.6.234.254:45473/ using document root <none> and password file <none>
I20260620 11:44:47.677609 19262 fs_manager.cc:359] Metadata directory not provided
I20260620 11:44:47.677651 19262 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 11:44:47.677850 19262 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Jun 20 11:44:47 dist-test-slave-cjc6 krb5kdc[18818](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781955887, etypes {rep=17 tkt=17 ses=17}, kudu/127.6.234.254@KRBTEST.COM for HTTP/127.6.234.212@KRBTEST.COM
I20260620 11:44:47.724239 19262 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task1JRg7N/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781955746769051-7083-0/minicluster-data/master-0/data/instance:
uuid: "cd8640322bce49ccb95296ca8860fadd"
format_stamp: "Formatted at 2026-06-20 11:44:47 on dist-test-slave-cjc6"
server_key: "fe09eddeb1340e17445e5da49debeeae"
server_key_iv: "6249587c155311cb859a4788807891aa"
server_key_version: "kuduclusterkey@0"
I20260620 11:44:47.724726 19262 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task1JRg7N/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781955746769051-7083-0/minicluster-data/master-0/wal/instance:
uuid: "cd8640322bce49ccb95296ca8860fadd"
format_stamp: "Formatted at 2026-06-20 11:44:47 on dist-test-slave-cjc6"
server_key: "fe09eddeb1340e17445e5da49debeeae"
server_key_iv: "6249587c155311cb859a4788807891aa"
server_key_version: "kuduclusterkey@0"
I20260620 11:44:47.726612 19262 fs_manager.cc:691] Time spent creating directory manager: real 0.002s user 0.003s sys 0.000s
I20260620 11:44:47.740242 19277 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 11:44:47.740449 19262 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260620 11:44:47.740541 19262 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task1JRg7N/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781955746769051-7083-0/minicluster-data/master-0/data,/tmp/dist-test-task1JRg7N/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781955746769051-7083-0/minicluster-data/master-0/wal
uuid: "cd8640322bce49ccb95296ca8860fadd"
format_stamp: "Formatted at 2026-06-20 11:44:47 on dist-test-slave-cjc6"
server_key: "fe09eddeb1340e17445e5da49debeeae"
server_key_iv: "6249587c155311cb859a4788807891aa"
server_key_version: "kuduclusterkey@0"
I20260620 11:44:47.740638 19262 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task1JRg7N/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781955746769051-7083-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task1JRg7N/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781955746769051-7083-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task1JRg7N/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781955746769051-7083-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 11:44:47.761641 19262 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 11:44:47.763744 19262 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 11:44:47.763916 19262 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 11:44:47.768759 19262 rpc_server.cc:307] RPC server started. Bound to: 127.6.234.254:36781
I20260620 11:44:47.768795 19329 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.6.234.254:36781 every 8 connection(s)
I20260620 11:44:47.769133 19262 server_base.cc:1166] Dumped server information to /tmp/dist-test-task1JRg7N/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781955746769051-7083-0/minicluster-data/master-0/data/info.pb
I20260620 11:44:47.769865 19330 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 11:44:47.770679 7083 external_mini_cluster.cc:1388] Started /tmp/dist-test-task1JRg7N/build/release/bin/kudu as pid 19262
I20260620 11:44:47.770792 7083 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task1JRg7N/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781955746769051-7083-0/minicluster-data/master-0/wal/instance
I20260620 11:44:47.770984 7083 external_mini_cluster.cc:1428] Setting key d423c7f49b1e243d6e74778eb7c1c484
I20260620 11:44:47.772467 19330 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P cd8640322bce49ccb95296ca8860fadd: Bootstrap starting.
I20260620 11:44:47.773281 19330 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P cd8640322bce49ccb95296ca8860fadd: Neither blocks nor log segments found. Creating new log.
I20260620 11:44:47.773609 19330 log.cc:824] T 00000000000000000000000000000000 P cd8640322bce49ccb95296ca8860fadd: Log is configured to *not* fsync() on all Append() calls
I20260620 11:44:47.774310 19330 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P cd8640322bce49ccb95296ca8860fadd: No bootstrap required, opened a new log
I20260620 11:44:47.776482 19330 raft_consensus.cc:348] T 00000000000000000000000000000000 P cd8640322bce49ccb95296ca8860fadd [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "cd8640322bce49ccb95296ca8860fadd" member_type: VOTER last_known_addr { host: "127.6.234.254" port: 36781 } }
I20260620 11:44:47.776638 19330 raft_consensus.cc:374] T 00000000000000000000000000000000 P cd8640322bce49ccb95296ca8860fadd [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 11:44:47.776700 19330 raft_consensus.cc:729] T 00000000000000000000000000000000 P cd8640322bce49ccb95296ca8860fadd [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: cd8640322bce49ccb95296ca8860fadd, State: Initialized, Role: FOLLOWER
I20260620 11:44:47.776858 19330 consensus_queue.cc:260] T 00000000000000000000000000000000 P cd8640322bce49ccb95296ca8860fadd [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "cd8640322bce49ccb95296ca8860fadd" member_type: VOTER last_known_addr { host: "127.6.234.254" port: 36781 } }
I20260620 11:44:47.776952 19330 raft_consensus.cc:388] T 00000000000000000000000000000000 P cd8640322bce49ccb95296ca8860fadd [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260620 11:44:47.777004 19330 raft_consensus.cc:482] T 00000000000000000000000000000000 P cd8640322bce49ccb95296ca8860fadd [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260620 11:44:47.777073 19330 raft_consensus.cc:3037] T 00000000000000000000000000000000 P cd8640322bce49ccb95296ca8860fadd [term 0 FOLLOWER]: Advancing to term 1
I20260620 11:44:47.777832 19330 raft_consensus.cc:504] T 00000000000000000000000000000000 P cd8640322bce49ccb95296ca8860fadd [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "cd8640322bce49ccb95296ca8860fadd" member_type: VOTER last_known_addr { host: "127.6.234.254" port: 36781 } }
I20260620 11:44:47.777948 19330 leader_election.cc:302] T 00000000000000000000000000000000 P cd8640322bce49ccb95296ca8860fadd [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: cd8640322bce49ccb95296ca8860fadd; no voters:
I20260620 11:44:47.778113 19330 leader_election.cc:288] T 00000000000000000000000000000000 P cd8640322bce49ccb95296ca8860fadd [CANDIDATE]: Term 1 election: Requested vote from peers
I20260620 11:44:47.778142 19335 raft_consensus.cc:2781] T 00000000000000000000000000000000 P cd8640322bce49ccb95296ca8860fadd [term 1 FOLLOWER]: Leader election won for term 1
I20260620 11:44:47.778355 19335 raft_consensus.cc:686] T 00000000000000000000000000000000 P cd8640322bce49ccb95296ca8860fadd [term 1 LEADER]: Becoming Leader. State: Replica: cd8640322bce49ccb95296ca8860fadd, State: Running, Role: LEADER
I20260620 11:44:47.778463 19330 sys_catalog.cc:564] T 00000000000000000000000000000000 P cd8640322bce49ccb95296ca8860fadd [sys.catalog]: configured and running, proceeding with master startup.
I20260620 11:44:47.778488 19335 consensus_queue.cc:237] T 00000000000000000000000000000000 P cd8640322bce49ccb95296ca8860fadd [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "cd8640322bce49ccb95296ca8860fadd" member_type: VOTER last_known_addr { host: "127.6.234.254" port: 36781 } }
I20260620 11:44:47.778861 19335 sys_catalog.cc:455] T 00000000000000000000000000000000 P cd8640322bce49ccb95296ca8860fadd [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "cd8640322bce49ccb95296ca8860fadd" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "cd8640322bce49ccb95296ca8860fadd" member_type: VOTER last_known_addr { host: "127.6.234.254" port: 36781 } } }
I20260620 11:44:47.779021 19335 sys_catalog.cc:458] T 00000000000000000000000000000000 P cd8640322bce49ccb95296ca8860fadd [sys.catalog]: This master's current role is: LEADER
Jun 20 11:44:47 dist-test-slave-cjc6 krb5kdc[18818](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781955887, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.6.234.254@KRBTEST.COM
I20260620 11:44:47.779218 19335 sys_catalog.cc:455] T 00000000000000000000000000000000 P cd8640322bce49ccb95296ca8860fadd [sys.catalog]: SysCatalogTable state changed. Reason: New leader cd8640322bce49ccb95296ca8860fadd. Latest consensus state: current_term: 1 leader_uuid: "cd8640322bce49ccb95296ca8860fadd" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "cd8640322bce49ccb95296ca8860fadd" member_type: VOTER last_known_addr { host: "127.6.234.254" port: 36781 } } }
I20260620 11:44:47.779279 19335 sys_catalog.cc:458] T 00000000000000000000000000000000 P cd8640322bce49ccb95296ca8860fadd [sys.catalog]: This master's current role is: LEADER
I20260620 11:44:47.779402 19343 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260620 11:44:47.779949 19343 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260620 11:44:47.780236 19330 ranger_client.cc:318] Using new properties file: /tmp/dist-test-task1JRg7N/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781955746769051-7083-0/minicluster-data/master-0/logs/kudu-ranger-subprocess-log4j2.properties
I20260620 11:44:47.786142 19333 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 11:44:47.771519 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58610 (local address 127.6.234.254:36781)
0620 11:44:47.771698 (+ 179us) server_negotiation.cc:206] Beginning negotiation
0620 11:44:47.771708 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0620 11:44:47.771743 (+ 35us) server_negotiation.cc:407] Connection header received
0620 11:44:47.772451 (+ 708us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 11:44:47.772461 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 11:44:47.772527 (+ 66us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 11:44:47.772684 (+ 157us) server_negotiation.cc:226] Negotiated authn=SASL
0620 11:44:47.773499 (+ 815us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 11:44:47.775015 (+ 1516us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 11:44:47.776115 (+ 1100us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 11:44:47.776729 (+ 614us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 11:44:47.777358 (+ 629us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 11:44:47.779833 (+ 2475us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 11:44:47.779838 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 11:44:47.779838 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 11:44:47.779850 (+ 12us) server_negotiation.cc:922] calling sasl_server_start()
0620 11:44:47.783807 (+ 3957us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 11:44:47.784324 (+ 517us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 11:44:47.784328 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 11:44:47.784328 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 11:44:47.784484 (+ 156us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 11:44:47.784720 (+ 236us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 11:44:47.784721 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 11:44:47.784721 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 11:44:47.785060 (+ 339us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 11:44:47.785670 (+ 610us) server_negotiation.cc:1035] Waiting for connection context
0620 11:44:47.785764 (+ 94us) server_negotiation.cc:299] Negotiation successful
0620 11:44:47.785921 (+ 157us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":136,"thread_start_us":72,"threads_started":1}
I20260620 11:44:47.787861 19343 catalog_manager.cc:1347] Generated new cluster ID: 29c25e96b0964232a66f293a469bd994
I20260620 11:44:47.787915 19343 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260620 11:44:47.806943 19343 catalog_manager.cc:1370] Generated new certificate authority record
I20260620 11:44:47.807518 19343 catalog_manager.cc:1504] Loading token signing keys...
I20260620 11:44:47.812351 19343 catalog_manager.cc:5868] T 00000000000000000000000000000000 P cd8640322bce49ccb95296ca8860fadd: Generated new TSK 0
I20260620 11:44:47.812528 19343 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.6.234.193@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.6.234.193@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.6.234.193 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task1JRg7N/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781955746769051-7083-0/krb5kdc/kudu_127.6.234.193.keytab.
Entry for principal kudu/127.6.234.193 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task1JRg7N/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781955746769051-7083-0/krb5kdc/kudu_127.6.234.193.keytab.
I20260620 11:44:48.476027 7083 external_mini_cluster.cc:1326] Running /tmp/dist-test-task1JRg7N/build/release/bin/kudu
/tmp/dist-test-task1JRg7N/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task1JRg7N/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781955746769051-7083-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-task1JRg7N/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781955746769051-7083-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task1JRg7N/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781955746769051-7083-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-task1JRg7N/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781955746769051-7083-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.6.234.212:44851/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.6.234.193:0
--local_ip_for_outbound_sockets=127.6.234.193
--webserver_interface=127.6.234.193
--webserver_port=0
--tserver_master_addrs=127.6.234.254:36781
--keytab_file=/tmp/dist-test-task1JRg7N/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781955746769051-7083-0/krb5kdc/kudu_127.6.234.193.keytab
--principal=kudu/127.6.234.193
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.6.234.212:38181
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task1JRg7N/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781955746769051-7083-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task1JRg7N/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781955746769051-7083-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task1JRg7N/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781955746769051-7083-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 11:44:48.577521 19375 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 11:44:48.577836 19375 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 11:44:48.577873 19375 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 11:44:48.579587 19375 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 11:44:48.579646 19375 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 11:44:48.579684 19375 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.6.234.193
I20260620 11:44:48.581811 19375 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.6.234.212:38181
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-task1JRg7N/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781955746769051-7083-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-task1JRg7N/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781955746769051-7083-0/minicluster-data/ts-0/wal
--ranger_kms_url=127.6.234.212:44851/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task1JRg7N/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781955746769051-7083-0/krb5kdc/kudu_127.6.234.193.keytab
--principal=kudu/127.6.234.193
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.6.234.193:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task1JRg7N/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781955746769051-7083-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.6.234.193
--webserver_port=0
--tserver_master_addrs=127.6.234.254:36781
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.19375
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.6.234.193
--log_dir=/tmp/dist-test-task1JRg7N/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781955746769051-7083-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 20 Jun 2026 10:32:37 UTC on ccycloud.quasar-bgyofv.root.comops.site
build id 4457173
I20260620 11:44:48.582053 19375 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 11:44:48.582319 19375 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 11:44:48.588447 19383 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 11:44:48.588443 19381 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 11:44:48.588533 19380 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 11:44:48.589489 19375 server_base.cc:1034] running on GCE node
I20260620 11:44:48.589762 19375 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 11:44:48.590036 19375 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 11:44:48.591249 19375 hybrid_clock.cc:648] HybridClock initialized: now 1781955888591233 us; error 27 us; skew 500 ppm
Jun 20 11:44:48 dist-test-slave-cjc6 krb5kdc[18818](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781955888, etypes {rep=17 tkt=17 ses=17}, kudu/127.6.234.193@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 11:44:48.594422 19375 init.cc:377] Logged in from keytab as kudu/127.6.234.193@KRBTEST.COM (short username kudu)
I20260620 11:44:48.594928 19375 webserver.cc:460] Webserver started at http://127.6.234.193:45539/ using document root <none> and password file <none>
I20260620 11:44:48.595193 19375 fs_manager.cc:359] Metadata directory not provided
I20260620 11:44:48.595255 19375 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 11:44:48.595402 19375 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Jun 20 11:44:48 dist-test-slave-cjc6 krb5kdc[18818](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781955888, etypes {rep=17 tkt=17 ses=17}, kudu/127.6.234.193@KRBTEST.COM for HTTP/127.6.234.212@KRBTEST.COM
I20260620 11:44:48.609382 19375 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task1JRg7N/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781955746769051-7083-0/minicluster-data/ts-0/data/instance:
uuid: "d6358116b2454b2fb815a367090682fd"
format_stamp: "Formatted at 2026-06-20 11:44:48 on dist-test-slave-cjc6"
server_key: "cf54ba694dd57f633007b50d582e9402"
server_key_iv: "162d6519aa2c5f5641e2861f21f342e2"
server_key_version: "kuduclusterkey@0"
I20260620 11:44:48.610081 19375 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task1JRg7N/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781955746769051-7083-0/minicluster-data/ts-0/wal/instance:
uuid: "d6358116b2454b2fb815a367090682fd"
format_stamp: "Formatted at 2026-06-20 11:44:48 on dist-test-slave-cjc6"
server_key: "cf54ba694dd57f633007b50d582e9402"
server_key_iv: "162d6519aa2c5f5641e2861f21f342e2"
server_key_version: "kuduclusterkey@0"
I20260620 11:44:48.611994 19375 fs_manager.cc:691] Time spent creating directory manager: real 0.002s user 0.000s sys 0.001s
I20260620 11:44:48.631225 19392 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 11:44:48.631424 19375 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.001s
I20260620 11:44:48.631531 19375 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task1JRg7N/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781955746769051-7083-0/minicluster-data/ts-0/data,/tmp/dist-test-task1JRg7N/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781955746769051-7083-0/minicluster-data/ts-0/wal
uuid: "d6358116b2454b2fb815a367090682fd"
format_stamp: "Formatted at 2026-06-20 11:44:48 on dist-test-slave-cjc6"
server_key: "cf54ba694dd57f633007b50d582e9402"
server_key_iv: "162d6519aa2c5f5641e2861f21f342e2"
server_key_version: "kuduclusterkey@0"
I20260620 11:44:48.631631 19375 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task1JRg7N/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781955746769051-7083-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-task1JRg7N/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781955746769051-7083-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-task1JRg7N/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781955746769051-7083-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 11:44:48.668764 19375 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 11:44:48.671276 19375 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 11:44:48.671496 19375 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 11:44:48.671778 19375 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 11:44:48.672183 19375 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 11:44:48.672269 19375 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 11:44:48.672379 19375 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 11:44:48.672453 19375 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 11:44:48.679075 19375 rpc_server.cc:307] RPC server started. Bound to: 127.6.234.193:38207
I20260620 11:44:48.679858 19375 server_base.cc:1166] Dumped server information to /tmp/dist-test-task1JRg7N/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781955746769051-7083-0/minicluster-data/ts-0/data/info.pb
I20260620 11:44:48.682683 19505 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.6.234.193:38207 every 8 connection(s)
I20260620 11:44:48.684537 7083 external_mini_cluster.cc:1388] Started /tmp/dist-test-task1JRg7N/build/release/bin/kudu as pid 19375
I20260620 11:44:48.684643 7083 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task1JRg7N/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781955746769051-7083-0/minicluster-data/ts-0/wal/instance
I20260620 11:44:48.684825 7083 external_mini_cluster.cc:1428] Setting key e57e904367ff55491a2d9f277204be28
Jun 20 11:44:48 dist-test-slave-cjc6 krb5kdc[18818](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781955888, etypes {rep=17 tkt=17 ses=17}, kudu/127.6.234.193@KRBTEST.COM for kudu/127.6.234.254@KRBTEST.COM
I20260620 11:44:48.694289 19509 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 11:44:48.683396 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.6.234.193:51491 (local address 127.6.234.254:36781)
0620 11:44:48.683525 (+ 129us) server_negotiation.cc:206] Beginning negotiation
0620 11:44:48.683527 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 11:44:48.683720 (+ 193us) server_negotiation.cc:407] Connection header received
0620 11:44:48.684440 (+ 720us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 11:44:48.684442 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 11:44:48.684456 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 11:44:48.684504 (+ 48us) server_negotiation.cc:226] Negotiated authn=SASL
0620 11:44:48.685319 (+ 815us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 11:44:48.686151 (+ 832us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 11:44:48.687637 (+ 1486us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 11:44:48.687951 (+ 314us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 11:44:48.687993 (+ 42us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 11:44:48.690610 (+ 2617us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 11:44:48.690612 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 11:44:48.690612 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 11:44:48.690619 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0620 11:44:48.692535 (+ 1916us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 11:44:48.693037 (+ 502us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 11:44:48.693038 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 11:44:48.693039 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 11:44:48.693103 (+ 64us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 11:44:48.693397 (+ 294us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 11:44:48.693398 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 11:44:48.693398 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 11:44:48.693546 (+ 148us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 11:44:48.693591 (+ 45us) server_negotiation.cc:1035] Waiting for connection context
0620 11:44:48.694092 (+ 501us) server_negotiation.cc:299] Negotiation successful
0620 11:44:48.694186 (+ 94us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":103,"thread_start_us":71,"threads_started":1}
I20260620 11:44:48.695098 19508 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 11:44:48.683363 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.234.254:36781 (local address 127.6.234.193:51491)
0620 11:44:48.683600 (+ 237us) negotiation.cc:107] Waiting for socket to connect
0620 11:44:48.683641 (+ 41us) client_negotiation.cc:174] Beginning negotiation
0620 11:44:48.684302 (+ 661us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 11:44:48.684695 (+ 393us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 11:44:48.684699 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 11:44:48.684933 (+ 234us) client_negotiation.cc:189] Negotiated authn=SASL
0620 11:44:48.685208 (+ 275us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 11:44:48.685214 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 11:44:48.686266 (+ 1052us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 11:44:48.686267 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 11:44:48.687536 (+ 1269us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 11:44:48.687538 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 11:44:48.688090 (+ 552us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 11:44:48.688090 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 11:44:48.688223 (+ 133us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 11:44:48.688823 (+ 600us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 11:44:48.688833 (+ 10us) client_negotiation.cc:656] calling sasl_client_start()
0620 11:44:48.690502 (+ 1669us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 11:44:48.692638 (+ 2136us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 11:44:48.692646 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 11:44:48.692648 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0620 11:44:48.692945 (+ 297us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 11:44:48.693174 (+ 229us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 11:44:48.693174 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 11:44:48.693175 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0620 11:44:48.693303 (+ 128us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 11:44:48.693640 (+ 337us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 11:44:48.693646 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 11:44:48.693929 (+ 283us) client_negotiation.cc:769] Sending connection context
0620 11:44:48.694652 (+ 723us) client_negotiation.cc:240] Negotiation successful
0620 11:44:48.694861 (+ 209us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":197,"thread_start_us":92,"threads_started":1}
I20260620 11:44:48.695892 19506 heartbeater.cc:347] Connected to a master server at 127.6.234.254:36781
I20260620 11:44:48.695986 19506 heartbeater.cc:464] Registering TS with master...
I20260620 11:44:48.696169 19506 heartbeater.cc:511] Master 127.6.234.254:36781 requested a full tablet report, sending...
I20260620 11:44:48.696750 19294 ts_manager.cc:194] Registered new tserver with Master: d6358116b2454b2fb815a367090682fd (127.6.234.193:38207)
I20260620 11:44:48.697657 19294 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.6.234.193@KRBTEST.COM'} at 127.6.234.193:51491
WARNING: no policy specified for kudu/127.6.234.194@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.6.234.194@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.6.234.194 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task1JRg7N/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781955746769051-7083-0/krb5kdc/kudu_127.6.234.194.keytab.
Entry for principal kudu/127.6.234.194 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task1JRg7N/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781955746769051-7083-0/krb5kdc/kudu_127.6.234.194.keytab.
I20260620 11:44:48.744110 7083 external_mini_cluster.cc:1326] Running /tmp/dist-test-task1JRg7N/build/release/bin/kudu
/tmp/dist-test-task1JRg7N/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task1JRg7N/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781955746769051-7083-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-task1JRg7N/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781955746769051-7083-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task1JRg7N/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781955746769051-7083-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-task1JRg7N/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781955746769051-7083-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.6.234.212:44851/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.6.234.194:0
--local_ip_for_outbound_sockets=127.6.234.194
--webserver_interface=127.6.234.194
--webserver_port=0
--tserver_master_addrs=127.6.234.254:36781
--keytab_file=/tmp/dist-test-task1JRg7N/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781955746769051-7083-0/krb5kdc/kudu_127.6.234.194.keytab
--principal=kudu/127.6.234.194
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.6.234.212:38181
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task1JRg7N/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781955746769051-7083-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task1JRg7N/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781955746769051-7083-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task1JRg7N/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781955746769051-7083-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Jun 20 11:44:48 dist-test-slave-cjc6 krb5kdc[18818](info): AS_REQ (1 etypes {17}) 127.0.0.1: ISSUE: authtime 1781955888, etypes {rep=17 tkt=17 ses=17}, kudu/127.6.234.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
W20260620 11:44:48.862731 19514 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 11:44:48.863035 19514 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 11:44:48.863072 19514 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 11:44:48.864727 19514 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 11:44:48.864840 19514 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 11:44:48.864900 19514 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.6.234.194
I20260620 11:44:48.867053 19514 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.6.234.212:38181
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-task1JRg7N/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781955746769051-7083-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-task1JRg7N/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781955746769051-7083-0/minicluster-data/ts-1/wal
--ranger_kms_url=127.6.234.212:44851/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task1JRg7N/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781955746769051-7083-0/krb5kdc/kudu_127.6.234.194.keytab
--principal=kudu/127.6.234.194
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.6.234.194:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task1JRg7N/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781955746769051-7083-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.6.234.194
--webserver_port=0
--tserver_master_addrs=127.6.234.254:36781
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.19514
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.6.234.194
--log_dir=/tmp/dist-test-task1JRg7N/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781955746769051-7083-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 20 Jun 2026 10:32:37 UTC on ccycloud.quasar-bgyofv.root.comops.site
build id 4457173
I20260620 11:44:48.867527 19514 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 11:44:48.867792 19514 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 11:44:48.873960 19521 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 11:44:48.873957 19523 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 11:44:48.874478 19520 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 11:44:48.874902 19514 server_base.cc:1034] running on GCE node
I20260620 11:44:48.875108 19514 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 11:44:48.875370 19514 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 11:44:48.876528 19514 hybrid_clock.cc:648] HybridClock initialized: now 1781955888876507 us; error 34 us; skew 500 ppm
Jun 20 11:44:48 dist-test-slave-cjc6 krb5kdc[18818](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781955888, etypes {rep=17 tkt=17 ses=17}, kudu/127.6.234.194@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 11:44:48.879776 19514 init.cc:377] Logged in from keytab as kudu/127.6.234.194@KRBTEST.COM (short username kudu)
I20260620 11:44:48.880393 19514 webserver.cc:460] Webserver started at http://127.6.234.194:39979/ using document root <none> and password file <none>
I20260620 11:44:48.880661 19514 fs_manager.cc:359] Metadata directory not provided
I20260620 11:44:48.880728 19514 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 11:44:48.880874 19514 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Jun 20 11:44:48 dist-test-slave-cjc6 krb5kdc[18818](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781955888, etypes {rep=17 tkt=17 ses=17}, kudu/127.6.234.194@KRBTEST.COM for HTTP/127.6.234.212@KRBTEST.COM
I20260620 11:44:48.898844 19514 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task1JRg7N/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781955746769051-7083-0/minicluster-data/ts-1/data/instance:
uuid: "c7d12946c3f1455fae82e6acc127fff2"
format_stamp: "Formatted at 2026-06-20 11:44:48 on dist-test-slave-cjc6"
server_key: "9fc030319f1b9ef95cc43fc7440b6d11"
server_key_iv: "778d0ab3e79c301c20dc90783d644c4a"
server_key_version: "kuduclusterkey@0"
I20260620 11:44:48.899339 19514 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task1JRg7N/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781955746769051-7083-0/minicluster-data/ts-1/wal/instance:
uuid: "c7d12946c3f1455fae82e6acc127fff2"
format_stamp: "Formatted at 2026-06-20 11:44:48 on dist-test-slave-cjc6"
server_key: "9fc030319f1b9ef95cc43fc7440b6d11"
server_key_iv: "778d0ab3e79c301c20dc90783d644c4a"
server_key_version: "kuduclusterkey@0"
I20260620 11:44:48.901265 19514 fs_manager.cc:691] Time spent creating directory manager: real 0.002s user 0.003s sys 0.001s
I20260620 11:44:48.913617 19530 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 11:44:48.913872 19514 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.000s
I20260620 11:44:48.913990 19514 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task1JRg7N/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781955746769051-7083-0/minicluster-data/ts-1/data,/tmp/dist-test-task1JRg7N/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781955746769051-7083-0/minicluster-data/ts-1/wal
uuid: "c7d12946c3f1455fae82e6acc127fff2"
format_stamp: "Formatted at 2026-06-20 11:44:48 on dist-test-slave-cjc6"
server_key: "9fc030319f1b9ef95cc43fc7440b6d11"
server_key_iv: "778d0ab3e79c301c20dc90783d644c4a"
server_key_version: "kuduclusterkey@0"
I20260620 11:44:48.914104 19514 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task1JRg7N/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781955746769051-7083-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-task1JRg7N/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781955746769051-7083-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-task1JRg7N/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781955746769051-7083-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 11:44:48.923758 19514 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 11:44:48.925863 19514 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 11:44:48.926009 19514 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 11:44:48.926258 19514 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 11:44:48.926604 19514 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 11:44:48.926646 19514 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 11:44:48.926694 19514 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 11:44:48.926717 19514 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 11:44:48.934116 19514 rpc_server.cc:307] RPC server started. Bound to: 127.6.234.194:36505
I20260620 11:44:48.934213 19643 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.6.234.194:36505 every 8 connection(s)
I20260620 11:44:48.934544 19514 server_base.cc:1166] Dumped server information to /tmp/dist-test-task1JRg7N/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781955746769051-7083-0/minicluster-data/ts-1/data/info.pb
Jun 20 11:44:48 dist-test-slave-cjc6 krb5kdc[18818](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781955888, etypes {rep=17 tkt=17 ses=17}, kudu/127.6.234.194@KRBTEST.COM for kudu/127.6.234.254@KRBTEST.COM
I20260620 11:44:48.942199 7083 external_mini_cluster.cc:1388] Started /tmp/dist-test-task1JRg7N/build/release/bin/kudu as pid 19514
I20260620 11:44:48.942327 7083 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task1JRg7N/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781955746769051-7083-0/minicluster-data/ts-1/wal/instance
I20260620 11:44:48.942548 7083 external_mini_cluster.cc:1428] Setting key b5ea1a1bb531b4d376ee15ed6e21473b
I20260620 11:44:48.946674 19646 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 11:44:48.935322 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.234.254:36781 (local address 127.6.234.194:34935)
0620 11:44:48.935470 (+ 148us) negotiation.cc:107] Waiting for socket to connect
0620 11:44:48.935512 (+ 42us) client_negotiation.cc:174] Beginning negotiation
0620 11:44:48.935991 (+ 479us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 11:44:48.936240 (+ 249us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 11:44:48.936243 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 11:44:48.936493 (+ 250us) client_negotiation.cc:189] Negotiated authn=SASL
0620 11:44:48.936682 (+ 189us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 11:44:48.936686 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 11:44:48.937800 (+ 1114us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 11:44:48.937802 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 11:44:48.938606 (+ 804us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 11:44:48.938609 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 11:44:48.939306 (+ 697us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 11:44:48.939307 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 11:44:48.939440 (+ 133us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 11:44:48.940338 (+ 898us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 11:44:48.940348 (+ 10us) client_negotiation.cc:656] calling sasl_client_start()
0620 11:44:48.942778 (+ 2430us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 11:44:48.944926 (+ 2148us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 11:44:48.944932 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 11:44:48.944935 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0620 11:44:48.945236 (+ 301us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 11:44:48.945498 (+ 262us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 11:44:48.945499 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 11:44:48.945499 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0620 11:44:48.945635 (+ 136us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 11:44:48.946018 (+ 383us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 11:44:48.946024 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 11:44:48.946183 (+ 159us) client_negotiation.cc:769] Sending connection context
0620 11:44:48.946292 (+ 109us) client_negotiation.cc:240] Negotiation successful
0620 11:44:48.946438 (+ 146us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":107,"thread_start_us":79,"threads_started":1}
I20260620 11:44:48.946995 19509 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 11:44:48.935346 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.6.234.194:34935 (local address 127.6.234.254:36781)
0620 11:44:48.935403 (+ 57us) server_negotiation.cc:206] Beginning negotiation
0620 11:44:48.935406 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0620 11:44:48.935609 (+ 203us) server_negotiation.cc:407] Connection header received
0620 11:44:48.936088 (+ 479us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 11:44:48.936089 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 11:44:48.936104 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 11:44:48.936187 (+ 83us) server_negotiation.cc:226] Negotiated authn=SASL
0620 11:44:48.936786 (+ 599us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 11:44:48.937606 (+ 820us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 11:44:48.938705 (+ 1099us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 11:44:48.939024 (+ 319us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 11:44:48.939102 (+ 78us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 11:44:48.942891 (+ 3789us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 11:44:48.942893 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 11:44:48.942893 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 11:44:48.942900 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0620 11:44:48.944652 (+ 1752us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 11:44:48.945352 (+ 700us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 11:44:48.945353 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 11:44:48.945353 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 11:44:48.945417 (+ 64us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 11:44:48.945755 (+ 338us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 11:44:48.945756 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 11:44:48.945757 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 11:44:48.945913 (+ 156us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 11:44:48.945987 (+ 74us) server_negotiation.cc:1035] Waiting for connection context
0620 11:44:48.946806 (+ 819us) server_negotiation.cc:299] Negotiation successful
0620 11:44:48.946911 (+ 105us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":24}
I20260620 11:44:48.948138 19644 heartbeater.cc:347] Connected to a master server at 127.6.234.254:36781
I20260620 11:44:48.948248 19644 heartbeater.cc:464] Registering TS with master...
I20260620 11:44:48.948484 19644 heartbeater.cc:511] Master 127.6.234.254:36781 requested a full tablet report, sending...
I20260620 11:44:48.948968 19294 ts_manager.cc:194] Registered new tserver with Master: c7d12946c3f1455fae82e6acc127fff2 (127.6.234.194:36505)
I20260620 11:44:48.949473 19294 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.6.234.194@KRBTEST.COM'} at 127.6.234.194:34935
WARNING: no policy specified for kudu/127.6.234.195@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.6.234.195@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.6.234.195 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task1JRg7N/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781955746769051-7083-0/krb5kdc/kudu_127.6.234.195.keytab.
Entry for principal kudu/127.6.234.195 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task1JRg7N/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781955746769051-7083-0/krb5kdc/kudu_127.6.234.195.keytab.
I20260620 11:44:48.972591 7083 external_mini_cluster.cc:1326] Running /tmp/dist-test-task1JRg7N/build/release/bin/kudu
/tmp/dist-test-task1JRg7N/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task1JRg7N/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781955746769051-7083-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-task1JRg7N/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781955746769051-7083-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task1JRg7N/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781955746769051-7083-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-task1JRg7N/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781955746769051-7083-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.6.234.212:44851/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.6.234.195:0
--local_ip_for_outbound_sockets=127.6.234.195
--webserver_interface=127.6.234.195
--webserver_port=0
--tserver_master_addrs=127.6.234.254:36781
--keytab_file=/tmp/dist-test-task1JRg7N/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781955746769051-7083-0/krb5kdc/kudu_127.6.234.195.keytab
--principal=kudu/127.6.234.195
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.6.234.212:38181
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task1JRg7N/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781955746769051-7083-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task1JRg7N/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781955746769051-7083-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task1JRg7N/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781955746769051-7083-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 11:44:49.083257 19649 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 11:44:49.083537 19649 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 11:44:49.083580 19649 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 11:44:49.085216 19649 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 11:44:49.085278 19649 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 11:44:49.085319 19649 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.6.234.195
I20260620 11:44:49.087466 19649 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.6.234.212:38181
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-task1JRg7N/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781955746769051-7083-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-task1JRg7N/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781955746769051-7083-0/minicluster-data/ts-2/wal
--ranger_kms_url=127.6.234.212:44851/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task1JRg7N/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781955746769051-7083-0/krb5kdc/kudu_127.6.234.195.keytab
--principal=kudu/127.6.234.195
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.6.234.195:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task1JRg7N/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781955746769051-7083-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.6.234.195
--webserver_port=0
--tserver_master_addrs=127.6.234.254:36781
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.19649
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.6.234.195
--log_dir=/tmp/dist-test-task1JRg7N/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781955746769051-7083-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 20 Jun 2026 10:32:37 UTC on ccycloud.quasar-bgyofv.root.comops.site
build id 4457173
I20260620 11:44:49.087718 19649 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 11:44:49.088027 19649 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 11:44:49.094169 19655 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 11:44:49.094174 19654 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 11:44:49.094452 19657 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 11:44:49.095497 19649 server_base.cc:1034] running on GCE node
I20260620 11:44:49.095693 19649 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 11:44:49.095971 19649 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 11:44:49.097148 19649 hybrid_clock.cc:648] HybridClock initialized: now 1781955889097119 us; error 43 us; skew 500 ppm
Jun 20 11:44:49 dist-test-slave-cjc6 krb5kdc[18818](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781955889, etypes {rep=17 tkt=17 ses=17}, kudu/127.6.234.195@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 11:44:49.104247 19649 init.cc:377] Logged in from keytab as kudu/127.6.234.195@KRBTEST.COM (short username kudu)
I20260620 11:44:49.104987 19649 webserver.cc:460] Webserver started at http://127.6.234.195:44257/ using document root <none> and password file <none>
I20260620 11:44:49.105449 19649 fs_manager.cc:359] Metadata directory not provided
I20260620 11:44:49.105607 19649 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 11:44:49.105902 19649 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Jun 20 11:44:49 dist-test-slave-cjc6 krb5kdc[18818](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781955889, etypes {rep=17 tkt=17 ses=17}, kudu/127.6.234.195@KRBTEST.COM for HTTP/127.6.234.212@KRBTEST.COM
I20260620 11:44:49.122771 19649 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task1JRg7N/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781955746769051-7083-0/minicluster-data/ts-2/data/instance:
uuid: "d7ee4c01f53e4ede9a24c5786ac0c0fa"
format_stamp: "Formatted at 2026-06-20 11:44:49 on dist-test-slave-cjc6"
server_key: "9efdf8cfaae799babe7151e34a688df5"
server_key_iv: "6e008a9b2b7f7e34836f3932db9d5482"
server_key_version: "kuduclusterkey@0"
I20260620 11:44:49.123211 19649 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task1JRg7N/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781955746769051-7083-0/minicluster-data/ts-2/wal/instance:
uuid: "d7ee4c01f53e4ede9a24c5786ac0c0fa"
format_stamp: "Formatted at 2026-06-20 11:44:49 on dist-test-slave-cjc6"
server_key: "9efdf8cfaae799babe7151e34a688df5"
server_key_iv: "6e008a9b2b7f7e34836f3932db9d5482"
server_key_version: "kuduclusterkey@0"
I20260620 11:44:49.128039 19649 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.002s sys 0.001s
I20260620 11:44:49.148711 19664 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 11:44:49.148895 19649 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.000s sys 0.000s
I20260620 11:44:49.148984 19649 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task1JRg7N/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781955746769051-7083-0/minicluster-data/ts-2/data,/tmp/dist-test-task1JRg7N/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781955746769051-7083-0/minicluster-data/ts-2/wal
uuid: "d7ee4c01f53e4ede9a24c5786ac0c0fa"
format_stamp: "Formatted at 2026-06-20 11:44:49 on dist-test-slave-cjc6"
server_key: "9efdf8cfaae799babe7151e34a688df5"
server_key_iv: "6e008a9b2b7f7e34836f3932db9d5482"
server_key_version: "kuduclusterkey@0"
I20260620 11:44:49.149083 19649 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task1JRg7N/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781955746769051-7083-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-task1JRg7N/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781955746769051-7083-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-task1JRg7N/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781955746769051-7083-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 11:44:49.166267 19649 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 11:44:49.168371 19649 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 11:44:49.168534 19649 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 11:44:49.168769 19649 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 11:44:49.169145 19649 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 11:44:49.169198 19649 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 11:44:49.169245 19649 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 11:44:49.169268 19649 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 11:44:49.176509 19649 rpc_server.cc:307] RPC server started. Bound to: 127.6.234.195:41895
I20260620 11:44:49.176860 19649 server_base.cc:1166] Dumped server information to /tmp/dist-test-task1JRg7N/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781955746769051-7083-0/minicluster-data/ts-2/data/info.pb
I20260620 11:44:49.177506 19777 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.6.234.195:41895 every 8 connection(s)
I20260620 11:44:49.178167 7083 external_mini_cluster.cc:1388] Started /tmp/dist-test-task1JRg7N/build/release/bin/kudu as pid 19649
I20260620 11:44:49.178285 7083 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task1JRg7N/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781955746769051-7083-0/minicluster-data/ts-2/wal/instance
I20260620 11:44:49.178463 7083 external_mini_cluster.cc:1428] Setting key b4d7d2e580cdb390945b7bc96042a7df
Jun 20 11:44:49 dist-test-slave-cjc6 krb5kdc[18818](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781955889, etypes {rep=17 tkt=17 ses=17}, kudu/127.6.234.195@KRBTEST.COM for kudu/127.6.234.254@KRBTEST.COM
I20260620 11:44:49.187074 19509 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 11:44:49.177892 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.6.234.195:44299 (local address 127.6.234.254:36781)
0620 11:44:49.177958 (+ 66us) server_negotiation.cc:206] Beginning negotiation
0620 11:44:49.177994 (+ 36us) server_negotiation.cc:399] Waiting for connection header
0620 11:44:49.178209 (+ 215us) server_negotiation.cc:407] Connection header received
0620 11:44:49.179066 (+ 857us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 11:44:49.179067 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 11:44:49.179080 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 11:44:49.179112 (+ 32us) server_negotiation.cc:226] Negotiated authn=SASL
0620 11:44:49.179898 (+ 786us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 11:44:49.180478 (+ 580us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 11:44:49.181467 (+ 989us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 11:44:49.181679 (+ 212us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 11:44:49.181793 (+ 114us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 11:44:49.183826 (+ 2033us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 11:44:49.183828 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 11:44:49.183829 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 11:44:49.183836 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0620 11:44:49.185443 (+ 1607us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 11:44:49.185962 (+ 519us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 11:44:49.185964 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 11:44:49.185964 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 11:44:49.186025 (+ 61us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 11:44:49.186318 (+ 293us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 11:44:49.186318 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 11:44:49.186319 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 11:44:49.186456 (+ 137us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 11:44:49.186501 (+ 45us) server_negotiation.cc:1035] Waiting for connection context
0620 11:44:49.186886 (+ 385us) server_negotiation.cc:299] Negotiation successful
0620 11:44:49.186975 (+ 89us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":42}
I20260620 11:44:49.187649 19780 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 11:44:49.177645 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.234.254:36781 (local address 127.6.234.195:44299)
0620 11:44:49.178067 (+ 422us) negotiation.cc:107] Waiting for socket to connect
0620 11:44:49.178118 (+ 51us) client_negotiation.cc:174] Beginning negotiation
0620 11:44:49.178913 (+ 795us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 11:44:49.179318 (+ 405us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 11:44:49.179322 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 11:44:49.179530 (+ 208us) client_negotiation.cc:189] Negotiated authn=SASL
0620 11:44:49.179814 (+ 284us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 11:44:49.179819 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 11:44:49.180569 (+ 750us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 11:44:49.180570 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 11:44:49.181364 (+ 794us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 11:44:49.181366 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 11:44:49.181865 (+ 499us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 11:44:49.181866 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 11:44:49.181963 (+ 97us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 11:44:49.182489 (+ 526us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 11:44:49.182497 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0620 11:44:49.183738 (+ 1241us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 11:44:49.185537 (+ 1799us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 11:44:49.185544 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 11:44:49.185546 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0620 11:44:49.185869 (+ 323us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 11:44:49.186102 (+ 233us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 11:44:49.186103 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 11:44:49.186103 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0620 11:44:49.186234 (+ 131us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 11:44:49.186546 (+ 312us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 11:44:49.186552 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 11:44:49.186706 (+ 154us) client_negotiation.cc:769] Sending connection context
0620 11:44:49.187177 (+ 471us) client_negotiation.cc:240] Negotiation successful
0620 11:44:49.187464 (+ 287us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":378,"thread_start_us":112,"threads_started":1}
I20260620 11:44:49.188205 19778 heartbeater.cc:347] Connected to a master server at 127.6.234.254:36781
I20260620 11:44:49.188300 19778 heartbeater.cc:464] Registering TS with master...
I20260620 11:44:49.188498 19778 heartbeater.cc:511] Master 127.6.234.254:36781 requested a full tablet report, sending...
I20260620 11:44:49.188889 19294 ts_manager.cc:194] Registered new tserver with Master: d7ee4c01f53e4ede9a24c5786ac0c0fa (127.6.234.195:41895)
I20260620 11:44:49.189273 19294 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.6.234.195@KRBTEST.COM'} at 127.6.234.195:44299
I20260620 11:44:49.190021 7083 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260620 11:44:49.190814 7083 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task1JRg7N/build/release/bin/kudu with pid 19375
I20260620 11:44:49.197141 7083 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task1JRg7N/build/release/bin/kudu with pid 19514
I20260620 11:44:49.202868 7083 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task1JRg7N/build/release/bin/kudu with pid 19649
I20260620 11:44:49.208258 7083 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task1JRg7N/build/release/bin/kudu with pid 19262
I20260620 11:44:49.213755 7083 external_mini_cluster.cc:1326] Running /tmp/dist-test-task1JRg7N/build/release/bin/kudu
/tmp/dist-test-task1JRg7N/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task1JRg7N/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781955746769051-7083-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task1JRg7N/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781955746769051-7083-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task1JRg7N/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781955746769051-7083-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task1JRg7N/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781955746769051-7083-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.6.234.254:36781
--webserver_interface=127.6.234.254
--webserver_port=45473
--keytab_file=/tmp/dist-test-task1JRg7N/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781955746769051-7083-0/krb5kdc/kudu_127.6.234.254.keytab
--principal=kudu/127.6.234.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.6.234.212:38181
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.6.234.254:36781
--ranger_config_path=/tmp/dist-test-task1JRg7N/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781955746769051-7083-0/minicluster-data/ranger-client
--trusted_user_acl=test-admin
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.6.234.212:44851/kms
--rpc_trace_negotiation
--ranger_kms_url=invalid.host:1234/kms,127.6.234.212:44851/kms with env {KRB5CCNAME=/tmp/dist-test-task1JRg7N/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781955746769051-7083-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task1JRg7N/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781955746769051-7083-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task1JRg7N/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781955746769051-7083-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 11:44:49.295085 19781 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 11:44:49.295320 19781 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 11:44:49.295365 19781 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 11:44:49.296514 19781 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 11:44:49.296581 19781 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 11:44:49.296605 19781 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 11:44:49.296633 19781 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 11:44:49.296664 19781 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 11:44:49.298210 19781 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.6.234.212:38181
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-task1JRg7N/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781955746769051-7083-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task1JRg7N/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781955746769051-7083-0/minicluster-data/master-0/wal
--ranger_kms_url=invalid.host:1234/kms,127.6.234.212:44851/kms
--trusted_user_acl=<redacted>
--ipki_ca_key_size=768
--master_addresses=127.6.234.254:36781
--ranger_config_path=/tmp/dist-test-task1JRg7N/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781955746769051-7083-0/minicluster-data/ranger-client
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task1JRg7N/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781955746769051-7083-0/krb5kdc/kudu_127.6.234.254.keytab
--principal=kudu/127.6.234.254
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.6.234.254:36781
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task1JRg7N/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781955746769051-7083-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.6.234.254
--webserver_port=45473
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.19781
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task1JRg7N/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781955746769051-7083-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 20 Jun 2026 10:32:37 UTC on ccycloud.quasar-bgyofv.root.comops.site
build id 4457173
I20260620 11:44:49.298480 19781 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 11:44:49.298692 19781 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 11:44:49.303426 19787 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 11:44:49.303452 19786 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 11:44:49.303426 19789 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 11:44:49.303624 19781 server_base.cc:1034] running on GCE node
I20260620 11:44:49.303941 19781 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 11:44:49.304220 19781 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 11:44:49.305358 19781 hybrid_clock.cc:648] HybridClock initialized: now 1781955889305342 us; error 33 us; skew 500 ppm
Jun 20 11:44:49 dist-test-slave-cjc6 krb5kdc[18818](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781955889, etypes {rep=17 tkt=17 ses=17}, kudu/127.6.234.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 11:44:49.307783 19781 init.cc:377] Logged in from keytab as kudu/127.6.234.254@KRBTEST.COM (short username kudu)
I20260620 11:44:49.308213 19781 webserver.cc:460] Webserver started at http://127.6.234.254:45473/ using document root <none> and password file <none>
I20260620 11:44:49.308429 19781 fs_manager.cc:359] Metadata directory not provided
I20260620 11:44:49.308480 19781 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
Jun 20 11:44:49 dist-test-slave-cjc6 krb5kdc[18818](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781955889, etypes {rep=17 tkt=17 ses=17}, kudu/127.6.234.254@KRBTEST.COM for HTTP/127.6.234.212@KRBTEST.COM
I20260620 11:44:49.348985 19781 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.001s sys 0.000s
I20260620 11:44:49.349870 19797 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 11:44:49.350071 19781 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260620 11:44:49.350174 19781 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task1JRg7N/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781955746769051-7083-0/minicluster-data/master-0/data,/tmp/dist-test-task1JRg7N/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781955746769051-7083-0/minicluster-data/master-0/wal
uuid: "cd8640322bce49ccb95296ca8860fadd"
format_stamp: "Formatted at 2026-06-20 11:44:47 on dist-test-slave-cjc6"
server_key: "fe09eddeb1340e17445e5da49debeeae"
server_key_iv: "6249587c155311cb859a4788807891aa"
server_key_version: "kuduclusterkey@0"
I20260620 11:44:49.350622 19781 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task1JRg7N/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781955746769051-7083-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task1JRg7N/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781955746769051-7083-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task1JRg7N/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781955746769051-7083-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 11:44:49.370265 19781 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 11:44:49.372344 19781 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 11:44:49.372502 19781 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 11:44:49.376797 19781 rpc_server.cc:307] RPC server started. Bound to: 127.6.234.254:36781
I20260620 11:44:49.376842 19849 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.6.234.254:36781 every 8 connection(s)
I20260620 11:44:49.377213 19781 server_base.cc:1166] Dumped server information to /tmp/dist-test-task1JRg7N/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781955746769051-7083-0/minicluster-data/master-0/data/info.pb
I20260620 11:44:49.378496 19850 sys_catalog.cc:263] Verifying existing consensus state
I20260620 11:44:49.379171 19850 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P cd8640322bce49ccb95296ca8860fadd: Bootstrap starting.
I20260620 11:44:49.380072 7083 external_mini_cluster.cc:1388] Started /tmp/dist-test-task1JRg7N/build/release/bin/kudu as pid 19781
I20260620 11:44:49.380350 7083 external_mini_cluster.cc:1326] Running /tmp/dist-test-task1JRg7N/build/release/bin/kudu
/tmp/dist-test-task1JRg7N/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task1JRg7N/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781955746769051-7083-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-task1JRg7N/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781955746769051-7083-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task1JRg7N/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781955746769051-7083-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-task1JRg7N/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781955746769051-7083-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.6.234.212:44851/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.6.234.193:38207
--local_ip_for_outbound_sockets=127.6.234.193
--tserver_master_addrs=127.6.234.254:36781
--webserver_port=45539
--webserver_interface=127.6.234.193
--keytab_file=/tmp/dist-test-task1JRg7N/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781955746769051-7083-0/krb5kdc/kudu_127.6.234.193.keytab
--principal=kudu/127.6.234.193
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.6.234.212:38181
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--ranger_kms_url=invalid.host:1234/kms,127.6.234.212:44851/kms with env {KRB5CCNAME=/tmp/dist-test-task1JRg7N/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781955746769051-7083-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task1JRg7N/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781955746769051-7083-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task1JRg7N/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781955746769051-7083-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260620 11:44:49.387574 19850 log.cc:824] T 00000000000000000000000000000000 P cd8640322bce49ccb95296ca8860fadd: Log is configured to *not* fsync() on all Append() calls
I20260620 11:44:49.389441 19850 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P cd8640322bce49ccb95296ca8860fadd: Bootstrap replayed 1/1 log segments. Stats: ops{read=4 overwritten=0 applied=4 ignored=0} inserts{seen=3 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260620 11:44:49.389854 19850 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P cd8640322bce49ccb95296ca8860fadd: Bootstrap complete.
I20260620 11:44:49.392542 19850 raft_consensus.cc:348] T 00000000000000000000000000000000 P cd8640322bce49ccb95296ca8860fadd [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "cd8640322bce49ccb95296ca8860fadd" member_type: VOTER last_known_addr { host: "127.6.234.254" port: 36781 } }
I20260620 11:44:49.392812 19850 raft_consensus.cc:729] T 00000000000000000000000000000000 P cd8640322bce49ccb95296ca8860fadd [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: cd8640322bce49ccb95296ca8860fadd, State: Initialized, Role: FOLLOWER
I20260620 11:44:49.393018 19850 consensus_queue.cc:260] T 00000000000000000000000000000000 P cd8640322bce49ccb95296ca8860fadd [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "cd8640322bce49ccb95296ca8860fadd" member_type: VOTER last_known_addr { host: "127.6.234.254" port: 36781 } }
I20260620 11:44:49.393115 19850 raft_consensus.cc:388] T 00000000000000000000000000000000 P cd8640322bce49ccb95296ca8860fadd [term 1 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260620 11:44:49.393170 19850 raft_consensus.cc:482] T 00000000000000000000000000000000 P cd8640322bce49ccb95296ca8860fadd [term 1 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260620 11:44:49.393221 19850 raft_consensus.cc:3037] T 00000000000000000000000000000000 P cd8640322bce49ccb95296ca8860fadd [term 1 FOLLOWER]: Advancing to term 2
I20260620 11:44:49.394321 19850 raft_consensus.cc:504] T 00000000000000000000000000000000 P cd8640322bce49ccb95296ca8860fadd [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "cd8640322bce49ccb95296ca8860fadd" member_type: VOTER last_known_addr { host: "127.6.234.254" port: 36781 } }
I20260620 11:44:49.394445 19850 leader_election.cc:302] T 00000000000000000000000000000000 P cd8640322bce49ccb95296ca8860fadd [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: cd8640322bce49ccb95296ca8860fadd; no voters:
I20260620 11:44:49.394663 19850 leader_election.cc:288] T 00000000000000000000000000000000 P cd8640322bce49ccb95296ca8860fadd [CANDIDATE]: Term 2 election: Requested vote from peers
I20260620 11:44:49.394704 19854 raft_consensus.cc:2781] T 00000000000000000000000000000000 P cd8640322bce49ccb95296ca8860fadd [term 2 FOLLOWER]: Leader election won for term 2
I20260620 11:44:49.394811 19854 raft_consensus.cc:686] T 00000000000000000000000000000000 P cd8640322bce49ccb95296ca8860fadd [term 2 LEADER]: Becoming Leader. State: Replica: cd8640322bce49ccb95296ca8860fadd, State: Running, Role: LEADER
I20260620 11:44:49.394912 19854 consensus_queue.cc:237] T 00000000000000000000000000000000 P cd8640322bce49ccb95296ca8860fadd [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 2, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "cd8640322bce49ccb95296ca8860fadd" member_type: VOTER last_known_addr { host: "127.6.234.254" port: 36781 } }
I20260620 11:44:49.394937 19850 sys_catalog.cc:564] T 00000000000000000000000000000000 P cd8640322bce49ccb95296ca8860fadd [sys.catalog]: configured and running, proceeding with master startup.
I20260620 11:44:49.395378 19856 sys_catalog.cc:455] T 00000000000000000000000000000000 P cd8640322bce49ccb95296ca8860fadd [sys.catalog]: SysCatalogTable state changed. Reason: New leader cd8640322bce49ccb95296ca8860fadd. Latest consensus state: current_term: 2 leader_uuid: "cd8640322bce49ccb95296ca8860fadd" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "cd8640322bce49ccb95296ca8860fadd" member_type: VOTER last_known_addr { host: "127.6.234.254" port: 36781 } } }
I20260620 11:44:49.395468 19856 sys_catalog.cc:458] T 00000000000000000000000000000000 P cd8640322bce49ccb95296ca8860fadd [sys.catalog]: This master's current role is: LEADER
I20260620 11:44:49.395715 19858 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260620 11:44:49.396230 19858 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260620 11:44:49.396344 19858 catalog_manager.cc:1259] Loaded cluster ID: 29c25e96b0964232a66f293a469bd994
I20260620 11:44:49.396371 19858 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260620 11:44:49.396656 19850 ranger_client.cc:318] Using overwritten properties file: /tmp/dist-test-task1JRg7N/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781955746769051-7083-0/minicluster-data/master-0/logs/kudu-ranger-subprocess-log4j2.properties
I20260620 11:44:49.397248 19855 sys_catalog.cc:455] T 00000000000000000000000000000000 P cd8640322bce49ccb95296ca8860fadd [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 2 leader_uuid: "cd8640322bce49ccb95296ca8860fadd" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "cd8640322bce49ccb95296ca8860fadd" member_type: VOTER last_known_addr { host: "127.6.234.254" port: 36781 } } }
I20260620 11:44:49.397373 19855 sys_catalog.cc:458] T 00000000000000000000000000000000 P cd8640322bce49ccb95296ca8860fadd [sys.catalog]: This master's current role is: LEADER
I20260620 11:44:49.397487 19858 catalog_manager.cc:1504] Loading token signing keys...
I20260620 11:44:49.397667 19858 catalog_manager.cc:5879] T 00000000000000000000000000000000 P cd8640322bce49ccb95296ca8860fadd: Loaded TSK: 0
I20260620 11:44:49.397936 19858 catalog_manager.cc:1514] Initializing in-progress tserver states...
W20260620 11:44:49.473186 19852 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 11:44:49.473479 19852 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 11:44:49.473519 19852 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 11:44:49.475286 19852 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 11:44:49.475343 19852 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 11:44:49.475386 19852 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.6.234.193
I20260620 11:44:49.477569 19852 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.6.234.212:38181
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-task1JRg7N/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781955746769051-7083-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-task1JRg7N/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781955746769051-7083-0/minicluster-data/ts-0/wal
--ranger_kms_url=invalid.host:1234/kms,127.6.234.212:44851/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task1JRg7N/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781955746769051-7083-0/krb5kdc/kudu_127.6.234.193.keytab
--principal=kudu/127.6.234.193
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.6.234.193:38207
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task1JRg7N/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781955746769051-7083-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.6.234.193
--webserver_port=45539
--tserver_master_addrs=127.6.234.254:36781
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.19852
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.6.234.193
--log_dir=/tmp/dist-test-task1JRg7N/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781955746769051-7083-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 20 Jun 2026 10:32:37 UTC on ccycloud.quasar-bgyofv.root.comops.site
build id 4457173
I20260620 11:44:49.477880 19852 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 11:44:49.478147 19852 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 11:44:49.484568 19888 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 11:44:49.484773 19890 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 11:44:49.484573 19887 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 11:44:49.485267 19852 server_base.cc:1034] running on GCE node
I20260620 11:44:49.485464 19852 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 11:44:49.485739 19852 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 11:44:49.489040 19852 hybrid_clock.cc:648] HybridClock initialized: now 1781955889489002 us; error 47 us; skew 500 ppm
Jun 20 11:44:49 dist-test-slave-cjc6 krb5kdc[18818](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781955889, etypes {rep=17 tkt=17 ses=17}, kudu/127.6.234.193@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 11:44:49.492391 19852 init.cc:377] Logged in from keytab as kudu/127.6.234.193@KRBTEST.COM (short username kudu)
I20260620 11:44:49.492933 19852 webserver.cc:460] Webserver started at http://127.6.234.193:45539/ using document root <none> and password file <none>
I20260620 11:44:49.493222 19852 fs_manager.cc:359] Metadata directory not provided
I20260620 11:44:49.493312 19852 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
Jun 20 11:44:49 dist-test-slave-cjc6 krb5kdc[18818](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781955889, etypes {rep=17 tkt=17 ses=17}, kudu/127.6.234.193@KRBTEST.COM for HTTP/127.6.234.212@KRBTEST.COM
I20260620 11:44:49.513047 19852 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.001s sys 0.000s
I20260620 11:44:49.514192 19898 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 11:44:49.514493 19852 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260620 11:44:49.514600 19852 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task1JRg7N/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781955746769051-7083-0/minicluster-data/ts-0/data,/tmp/dist-test-task1JRg7N/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781955746769051-7083-0/minicluster-data/ts-0/wal
uuid: "d6358116b2454b2fb815a367090682fd"
format_stamp: "Formatted at 2026-06-20 11:44:48 on dist-test-slave-cjc6"
server_key: "cf54ba694dd57f633007b50d582e9402"
server_key_iv: "162d6519aa2c5f5641e2861f21f342e2"
server_key_version: "kuduclusterkey@0"
I20260620 11:44:49.515066 19852 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task1JRg7N/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781955746769051-7083-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-task1JRg7N/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781955746769051-7083-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-task1JRg7N/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781955746769051-7083-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 11:44:49.560144 19852 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 11:44:49.562572 19852 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 11:44:49.562723 19852 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 11:44:49.562986 19852 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 11:44:49.563331 19852 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 11:44:49.563381 19852 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 11:44:49.563413 19852 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 11:44:49.563438 19852 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 11:44:49.570209 19852 rpc_server.cc:307] RPC server started. Bound to: 127.6.234.193:38207
I20260620 11:44:49.570621 19852 server_base.cc:1166] Dumped server information to /tmp/dist-test-task1JRg7N/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781955746769051-7083-0/minicluster-data/ts-0/data/info.pb
I20260620 11:44:49.570619 20011 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.6.234.193:38207 every 8 connection(s)
I20260620 11:44:49.572872 7083 external_mini_cluster.cc:1388] Started /tmp/dist-test-task1JRg7N/build/release/bin/kudu as pid 19852
I20260620 11:44:49.573186 7083 external_mini_cluster.cc:1326] Running /tmp/dist-test-task1JRg7N/build/release/bin/kudu
/tmp/dist-test-task1JRg7N/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task1JRg7N/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781955746769051-7083-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-task1JRg7N/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781955746769051-7083-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task1JRg7N/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781955746769051-7083-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-task1JRg7N/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781955746769051-7083-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.6.234.212:44851/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.6.234.194:36505
--local_ip_for_outbound_sockets=127.6.234.194
--tserver_master_addrs=127.6.234.254:36781
--webserver_port=39979
--webserver_interface=127.6.234.194
--keytab_file=/tmp/dist-test-task1JRg7N/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781955746769051-7083-0/krb5kdc/kudu_127.6.234.194.keytab
--principal=kudu/127.6.234.194
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.6.234.212:38181
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--ranger_kms_url=invalid.host:1234/kms,127.6.234.212:44851/kms with env {KRB5CCNAME=/tmp/dist-test-task1JRg7N/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781955746769051-7083-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task1JRg7N/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781955746769051-7083-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task1JRg7N/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781955746769051-7083-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Jun 20 11:44:49 dist-test-slave-cjc6 krb5kdc[18818](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781955889, etypes {rep=17 tkt=17 ses=17}, kudu/127.6.234.193@KRBTEST.COM for kudu/127.6.234.254@KRBTEST.COM
I20260620 11:44:49.583933 20014 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 11:44:49.571753 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.234.254:36781 (local address 127.6.234.193:46523)
0620 11:44:49.571981 (+ 228us) negotiation.cc:107] Waiting for socket to connect
0620 11:44:49.572018 (+ 37us) client_negotiation.cc:174] Beginning negotiation
0620 11:44:49.572589 (+ 571us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 11:44:49.573243 (+ 654us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 11:44:49.573248 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 11:44:49.573462 (+ 214us) client_negotiation.cc:189] Negotiated authn=SASL
0620 11:44:49.573760 (+ 298us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 11:44:49.573765 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 11:44:49.575513 (+ 1748us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 11:44:49.575514 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 11:44:49.576810 (+ 1296us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 11:44:49.576814 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 11:44:49.577596 (+ 782us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 11:44:49.577597 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 11:44:49.577769 (+ 172us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 11:44:49.578352 (+ 583us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 11:44:49.578364 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0620 11:44:49.580096 (+ 1732us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 11:44:49.582289 (+ 2193us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 11:44:49.582295 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 11:44:49.582297 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0620 11:44:49.582561 (+ 264us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 11:44:49.582818 (+ 257us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 11:44:49.582819 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 11:44:49.582819 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0620 11:44:49.582929 (+ 110us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 11:44:49.583337 (+ 408us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 11:44:49.583342 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 11:44:49.583476 (+ 134us) client_negotiation.cc:769] Sending connection context
0620 11:44:49.583554 (+ 78us) client_negotiation.cc:240] Negotiation successful
0620 11:44:49.583697 (+ 143us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":190,"thread_start_us":78,"threads_started":1}
I20260620 11:44:49.584945 20015 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 11:44:49.571840 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.6.234.193:46523 (local address 127.6.234.254:36781)
0620 11:44:49.572812 (+ 972us) server_negotiation.cc:206] Beginning negotiation
0620 11:44:49.572822 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0620 11:44:49.572859 (+ 37us) server_negotiation.cc:407] Connection header received
0620 11:44:49.572998 (+ 139us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 11:44:49.573003 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 11:44:49.573056 (+ 53us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 11:44:49.573892 (+ 836us) server_negotiation.cc:226] Negotiated authn=SASL
0620 11:44:49.573988 (+ 96us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 11:44:49.575394 (+ 1406us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 11:44:49.576917 (+ 1523us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 11:44:49.577469 (+ 552us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 11:44:49.578035 (+ 566us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 11:44:49.580215 (+ 2180us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 11:44:49.580219 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 11:44:49.580220 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 11:44:49.580231 (+ 11us) server_negotiation.cc:922] calling sasl_server_start()
0620 11:44:49.582192 (+ 1961us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 11:44:49.582645 (+ 453us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 11:44:49.582647 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 11:44:49.582648 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 11:44:49.582736 (+ 88us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 11:44:49.582994 (+ 258us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 11:44:49.582994 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 11:44:49.582995 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 11:44:49.583261 (+ 266us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 11:44:49.584538 (+ 1277us) server_negotiation.cc:1035] Waiting for connection context
0620 11:44:49.584614 (+ 76us) server_negotiation.cc:299] Negotiation successful
0620 11:44:49.584734 (+ 120us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":914,"thread_start_us":102,"threads_started":1}
I20260620 11:44:49.585750 20012 heartbeater.cc:347] Connected to a master server at 127.6.234.254:36781
I20260620 11:44:49.585839 20012 heartbeater.cc:464] Registering TS with master...
I20260620 11:44:49.586014 20012 heartbeater.cc:511] Master 127.6.234.254:36781 requested a full tablet report, sending...
W20260620 11:44:49.586818 20012 heartbeater.cc:644] Failed to heartbeat to 127.6.234.254:36781 (0 consecutive failures): Service unavailable: Catalog manager is not initialized. State: Starting
I20260620 11:44:49.586890 20012 heartbeater.cc:464] Registering TS with master...
I20260620 11:44:49.586957 20012 heartbeater.cc:511] Master 127.6.234.254:36781 requested a full tablet report, sending...
I20260620 11:44:49.587342 20012 heartbeater.cc:464] Registering TS with master...
I20260620 11:44:49.587432 20012 heartbeater.cc:511] Master 127.6.234.254:36781 requested a full tablet report, sending...
W20260620 11:44:49.587785 20012 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
W20260620 11:44:49.690510 20016 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 11:44:49.690791 20016 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 11:44:49.690824 20016 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 11:44:49.692158 20016 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 11:44:49.692198 20016 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 11:44:49.692224 20016 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.6.234.194
I20260620 11:44:49.693933 20016 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.6.234.212:38181
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-task1JRg7N/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781955746769051-7083-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-task1JRg7N/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781955746769051-7083-0/minicluster-data/ts-1/wal
--ranger_kms_url=invalid.host:1234/kms,127.6.234.212:44851/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task1JRg7N/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781955746769051-7083-0/krb5kdc/kudu_127.6.234.194.keytab
--principal=kudu/127.6.234.194
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.6.234.194:36505
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task1JRg7N/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781955746769051-7083-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.6.234.194
--webserver_port=39979
--tserver_master_addrs=127.6.234.254:36781
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.20016
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.6.234.194
--log_dir=/tmp/dist-test-task1JRg7N/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781955746769051-7083-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 20 Jun 2026 10:32:37 UTC on ccycloud.quasar-bgyofv.root.comops.site
build id 4457173
I20260620 11:44:49.694176 20016 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 11:44:49.694443 20016 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 11:44:49.700188 20022 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 11:44:49.700256 20021 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 11:44:49.700469 20024 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 11:44:49.700733 20016 server_base.cc:1034] running on GCE node
I20260620 11:44:49.701087 20016 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 11:44:49.701362 20016 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 11:44:49.702513 20016 hybrid_clock.cc:648] HybridClock initialized: now 1781955889702491 us; error 39 us; skew 500 ppm
Jun 20 11:44:49 dist-test-slave-cjc6 krb5kdc[18818](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781955889, etypes {rep=17 tkt=17 ses=17}, kudu/127.6.234.194@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 11:44:49.705351 20016 init.cc:377] Logged in from keytab as kudu/127.6.234.194@KRBTEST.COM (short username kudu)
I20260620 11:44:49.705935 20016 webserver.cc:460] Webserver started at http://127.6.234.194:39979/ using document root <none> and password file <none>
I20260620 11:44:49.706210 20016 fs_manager.cc:359] Metadata directory not provided
I20260620 11:44:49.706288 20016 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
Jun 20 11:44:49 dist-test-slave-cjc6 krb5kdc[18818](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781955889, etypes {rep=17 tkt=17 ses=17}, kudu/127.6.234.194@KRBTEST.COM for HTTP/127.6.234.212@KRBTEST.COM
I20260620 11:44:49.723317 20016 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.000s sys 0.001s
I20260620 11:44:49.724418 20032 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 11:44:49.724627 20016 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260620 11:44:49.724740 20016 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task1JRg7N/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781955746769051-7083-0/minicluster-data/ts-1/data,/tmp/dist-test-task1JRg7N/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781955746769051-7083-0/minicluster-data/ts-1/wal
uuid: "c7d12946c3f1455fae82e6acc127fff2"
format_stamp: "Formatted at 2026-06-20 11:44:48 on dist-test-slave-cjc6"
server_key: "9fc030319f1b9ef95cc43fc7440b6d11"
server_key_iv: "778d0ab3e79c301c20dc90783d644c4a"
server_key_version: "kuduclusterkey@0"
I20260620 11:44:49.725227 20016 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task1JRg7N/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781955746769051-7083-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-task1JRg7N/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781955746769051-7083-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-task1JRg7N/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781955746769051-7083-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 11:44:49.763307 20016 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 11:44:49.765830 20016 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 11:44:49.766002 20016 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 11:44:49.766292 20016 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 11:44:49.766675 20016 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 11:44:49.766728 20016 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 11:44:49.766764 20016 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 11:44:49.766791 20016 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 11:44:49.774638 20016 rpc_server.cc:307] RPC server started. Bound to: 127.6.234.194:36505
I20260620 11:44:49.774713 20145 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.6.234.194:36505 every 8 connection(s)
I20260620 11:44:49.775154 20016 server_base.cc:1166] Dumped server information to /tmp/dist-test-task1JRg7N/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781955746769051-7083-0/minicluster-data/ts-1/data/info.pb
Jun 20 11:44:49 dist-test-slave-cjc6 krb5kdc[18818](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781955889, etypes {rep=17 tkt=17 ses=17}, kudu/127.6.234.194@KRBTEST.COM for kudu/127.6.234.254@KRBTEST.COM
I20260620 11:44:49.782766 7083 external_mini_cluster.cc:1388] Started /tmp/dist-test-task1JRg7N/build/release/bin/kudu as pid 20016
I20260620 11:44:49.783073 7083 external_mini_cluster.cc:1326] Running /tmp/dist-test-task1JRg7N/build/release/bin/kudu
/tmp/dist-test-task1JRg7N/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task1JRg7N/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781955746769051-7083-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-task1JRg7N/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781955746769051-7083-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task1JRg7N/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781955746769051-7083-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-task1JRg7N/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781955746769051-7083-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.6.234.212:44851/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.6.234.195:41895
--local_ip_for_outbound_sockets=127.6.234.195
--tserver_master_addrs=127.6.234.254:36781
--webserver_port=44257
--webserver_interface=127.6.234.195
--keytab_file=/tmp/dist-test-task1JRg7N/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781955746769051-7083-0/krb5kdc/kudu_127.6.234.195.keytab
--principal=kudu/127.6.234.195
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.6.234.212:38181
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--ranger_kms_url=invalid.host:1234/kms,127.6.234.212:44851/kms with env {KRB5CCNAME=/tmp/dist-test-task1JRg7N/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781955746769051-7083-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task1JRg7N/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781955746769051-7083-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task1JRg7N/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781955746769051-7083-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260620 11:44:49.787019 20015 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 11:44:49.775985 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.6.234.194:40117 (local address 127.6.234.254:36781)
0620 11:44:49.776040 (+ 55us) server_negotiation.cc:206] Beginning negotiation
0620 11:44:49.776042 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 11:44:49.776271 (+ 229us) server_negotiation.cc:407] Connection header received
0620 11:44:49.776818 (+ 547us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 11:44:49.776820 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 11:44:49.776833 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 11:44:49.776884 (+ 51us) server_negotiation.cc:226] Negotiated authn=SASL
0620 11:44:49.777463 (+ 579us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 11:44:49.778126 (+ 663us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 11:44:49.779163 (+ 1037us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 11:44:49.779378 (+ 215us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 11:44:49.779461 (+ 83us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 11:44:49.781644 (+ 2183us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 11:44:49.781646 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 11:44:49.781646 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 11:44:49.781653 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0620 11:44:49.785291 (+ 3638us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 11:44:49.785855 (+ 564us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 11:44:49.785857 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 11:44:49.785857 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 11:44:49.785920 (+ 63us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 11:44:49.786228 (+ 308us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 11:44:49.786229 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 11:44:49.786230 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 11:44:49.786371 (+ 141us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 11:44:49.786416 (+ 45us) server_negotiation.cc:1035] Waiting for connection context
0620 11:44:49.786825 (+ 409us) server_negotiation.cc:299] Negotiation successful
0620 11:44:49.786916 (+ 91us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":27}
I20260620 11:44:49.787572 20148 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 11:44:49.775952 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.234.254:36781 (local address 127.6.234.194:40117)
0620 11:44:49.776134 (+ 182us) negotiation.cc:107] Waiting for socket to connect
0620 11:44:49.776172 (+ 38us) client_negotiation.cc:174] Beginning negotiation
0620 11:44:49.776685 (+ 513us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 11:44:49.776956 (+ 271us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 11:44:49.776960 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 11:44:49.777150 (+ 190us) client_negotiation.cc:189] Negotiated authn=SASL
0620 11:44:49.777349 (+ 199us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 11:44:49.777353 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 11:44:49.778225 (+ 872us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 11:44:49.778226 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 11:44:49.779058 (+ 832us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 11:44:49.779060 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 11:44:49.779483 (+ 423us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 11:44:49.779484 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 11:44:49.779600 (+ 116us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 11:44:49.780158 (+ 558us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 11:44:49.780167 (+ 9us) client_negotiation.cc:656] calling sasl_client_start()
0620 11:44:49.781527 (+ 1360us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 11:44:49.785390 (+ 3863us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 11:44:49.785397 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 11:44:49.785400 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0620 11:44:49.785761 (+ 361us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 11:44:49.786001 (+ 240us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 11:44:49.786002 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 11:44:49.786002 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0620 11:44:49.786144 (+ 142us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 11:44:49.786466 (+ 322us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 11:44:49.786473 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 11:44:49.786640 (+ 167us) client_negotiation.cc:769] Sending connection context
0620 11:44:49.787163 (+ 523us) client_negotiation.cc:240] Negotiation successful
0620 11:44:49.787329 (+ 166us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":135,"thread_start_us":85,"threads_started":1}
I20260620 11:44:49.788054 20146 heartbeater.cc:347] Connected to a master server at 127.6.234.254:36781
I20260620 11:44:49.788164 20146 heartbeater.cc:464] Registering TS with master...
I20260620 11:44:49.788360 20146 heartbeater.cc:511] Master 127.6.234.254:36781 requested a full tablet report, sending...
W20260620 11:44:49.789042 20146 heartbeater.cc:644] Failed to heartbeat to 127.6.234.254:36781 (0 consecutive failures): Service unavailable: Catalog manager is not initialized. State: Starting
I20260620 11:44:49.789127 20146 heartbeater.cc:464] Registering TS with master...
I20260620 11:44:49.789199 20146 heartbeater.cc:511] Master 127.6.234.254:36781 requested a full tablet report, sending...
I20260620 11:44:49.789688 20146 heartbeater.cc:464] Registering TS with master...
I20260620 11:44:49.789798 20146 heartbeater.cc:511] Master 127.6.234.254:36781 requested a full tablet report, sending...
W20260620 11:44:49.790150 20146 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
W20260620 11:44:49.905586 20149 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 11:44:49.905903 20149 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 11:44:49.905941 20149 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 11:44:49.907641 20149 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 11:44:49.907706 20149 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 11:44:49.907747 20149 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.6.234.195
I20260620 11:44:49.909875 20149 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.6.234.212:38181
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-task1JRg7N/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781955746769051-7083-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-task1JRg7N/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781955746769051-7083-0/minicluster-data/ts-2/wal
--ranger_kms_url=invalid.host:1234/kms,127.6.234.212:44851/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task1JRg7N/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781955746769051-7083-0/krb5kdc/kudu_127.6.234.195.keytab
--principal=kudu/127.6.234.195
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.6.234.195:41895
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task1JRg7N/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781955746769051-7083-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.6.234.195
--webserver_port=44257
--tserver_master_addrs=127.6.234.254:36781
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.20149
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.6.234.195
--log_dir=/tmp/dist-test-task1JRg7N/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781955746769051-7083-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 20 Jun 2026 10:32:37 UTC on ccycloud.quasar-bgyofv.root.comops.site
build id 4457173
I20260620 11:44:49.910111 20149 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 11:44:49.910367 20149 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 11:44:49.916621 20158 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 11:44:49.916978 20149 server_base.cc:1034] running on GCE node
W20260620 11:44:49.916680 20156 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 11:44:49.916911 20155 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 11:44:49.917362 20149 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 11:44:49.917630 20149 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 11:44:49.918820 20149 hybrid_clock.cc:648] HybridClock initialized: now 1781955889918786 us; error 47 us; skew 500 ppm
Jun 20 11:44:49 dist-test-slave-cjc6 krb5kdc[18818](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781955889, etypes {rep=17 tkt=17 ses=17}, kudu/127.6.234.195@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 11:44:49.922049 20149 init.cc:377] Logged in from keytab as kudu/127.6.234.195@KRBTEST.COM (short username kudu)
I20260620 11:44:49.922535 20149 webserver.cc:460] Webserver started at http://127.6.234.195:44257/ using document root <none> and password file <none>
I20260620 11:44:49.922786 20149 fs_manager.cc:359] Metadata directory not provided
I20260620 11:44:49.922847 20149 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
Jun 20 11:44:49 dist-test-slave-cjc6 krb5kdc[18818](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781955889, etypes {rep=17 tkt=17 ses=17}, kudu/127.6.234.195@KRBTEST.COM for HTTP/127.6.234.212@KRBTEST.COM
I20260620 11:44:49.942121 20149 fs_manager.cc:709] Time spent opening directory manager: real 0.005s user 0.001s sys 0.000s
I20260620 11:44:49.943293 20166 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 11:44:49.943506 20149 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.000s
I20260620 11:44:49.943588 20149 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task1JRg7N/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781955746769051-7083-0/minicluster-data/ts-2/data,/tmp/dist-test-task1JRg7N/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781955746769051-7083-0/minicluster-data/ts-2/wal
uuid: "d7ee4c01f53e4ede9a24c5786ac0c0fa"
format_stamp: "Formatted at 2026-06-20 11:44:49 on dist-test-slave-cjc6"
server_key: "9efdf8cfaae799babe7151e34a688df5"
server_key_iv: "6e008a9b2b7f7e34836f3932db9d5482"
server_key_version: "kuduclusterkey@0"
I20260620 11:44:49.944047 20149 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task1JRg7N/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781955746769051-7083-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-task1JRg7N/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781955746769051-7083-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-task1JRg7N/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781955746769051-7083-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 11:44:49.966300 20149 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 11:44:49.968696 20149 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 11:44:49.968881 20149 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 11:44:49.969159 20149 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 11:44:49.969616 20149 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 11:44:49.969671 20149 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 11:44:49.969738 20149 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 11:44:49.969765 20149 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 11:44:49.976923 20149 rpc_server.cc:307] RPC server started. Bound to: 127.6.234.195:41895
I20260620 11:44:49.977093 20279 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.6.234.195:41895 every 8 connection(s)
I20260620 11:44:49.977377 20149 server_base.cc:1166] Dumped server information to /tmp/dist-test-task1JRg7N/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781955746769051-7083-0/minicluster-data/ts-2/data/info.pb
I20260620 11:44:49.979738 7083 external_mini_cluster.cc:1388] Started /tmp/dist-test-task1JRg7N/build/release/bin/kudu as pid 20149
Jun 20 11:44:49 dist-test-slave-cjc6 krb5kdc[18818](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781955889, etypes {rep=17 tkt=17 ses=17}, kudu/127.6.234.195@KRBTEST.COM for kudu/127.6.234.254@KRBTEST.COM
I20260620 11:44:49.987952 20282 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 11:44:49.978253 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.234.254:36781 (local address 127.6.234.195:60069)
0620 11:44:49.978423 (+ 170us) negotiation.cc:107] Waiting for socket to connect
0620 11:44:49.978465 (+ 42us) client_negotiation.cc:174] Beginning negotiation
0620 11:44:49.978984 (+ 519us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 11:44:49.979287 (+ 303us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 11:44:49.979291 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 11:44:49.979462 (+ 171us) client_negotiation.cc:189] Negotiated authn=SASL
0620 11:44:49.979776 (+ 314us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 11:44:49.979782 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 11:44:49.980621 (+ 839us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 11:44:49.980622 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 11:44:49.981481 (+ 859us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 11:44:49.981483 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 11:44:49.982083 (+ 600us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 11:44:49.982084 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 11:44:49.982221 (+ 137us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 11:44:49.982819 (+ 598us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 11:44:49.982828 (+ 9us) client_negotiation.cc:656] calling sasl_client_start()
0620 11:44:49.984439 (+ 1611us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 11:44:49.986299 (+ 1860us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 11:44:49.986306 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 11:44:49.986308 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0620 11:44:49.986605 (+ 297us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 11:44:49.986827 (+ 222us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 11:44:49.986828 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 11:44:49.986828 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0620 11:44:49.986972 (+ 144us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 11:44:49.987264 (+ 292us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 11:44:49.987270 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 11:44:49.987431 (+ 161us) client_negotiation.cc:769] Sending connection context
0620 11:44:49.987542 (+ 111us) client_negotiation.cc:240] Negotiation successful
0620 11:44:49.987695 (+ 153us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":127,"thread_start_us":80,"threads_started":1}
I20260620 11:44:49.988447 20015 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 11:44:49.978217 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.6.234.195:60069 (local address 127.6.234.254:36781)
0620 11:44:49.978266 (+ 49us) server_negotiation.cc:206] Beginning negotiation
0620 11:44:49.978268 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 11:44:49.978556 (+ 288us) server_negotiation.cc:407] Connection header received
0620 11:44:49.979108 (+ 552us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 11:44:49.979110 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 11:44:49.979124 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 11:44:49.979227 (+ 103us) server_negotiation.cc:226] Negotiated authn=SASL
0620 11:44:49.979906 (+ 679us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 11:44:49.980528 (+ 622us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 11:44:49.981582 (+ 1054us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 11:44:49.981924 (+ 342us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 11:44:49.981999 (+ 75us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 11:44:49.984539 (+ 2540us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 11:44:49.984540 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 11:44:49.984541 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 11:44:49.984548 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0620 11:44:49.986205 (+ 1657us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 11:44:49.986696 (+ 491us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 11:44:49.986697 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 11:44:49.986697 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 11:44:49.986756 (+ 59us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 11:44:49.987051 (+ 295us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 11:44:49.987052 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 11:44:49.987052 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 11:44:49.987180 (+ 128us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 11:44:49.987239 (+ 59us) server_negotiation.cc:1035] Waiting for connection context
0620 11:44:49.988276 (+ 1037us) server_negotiation.cc:299] Negotiation successful
0620 11:44:49.988357 (+ 81us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":23}
I20260620 11:44:49.989148 20280 heartbeater.cc:347] Connected to a master server at 127.6.234.254:36781
I20260620 11:44:49.989244 20280 heartbeater.cc:464] Registering TS with master...
I20260620 11:44:49.989449 20280 heartbeater.cc:511] Master 127.6.234.254:36781 requested a full tablet report, sending...
W20260620 11:44:49.990165 20280 heartbeater.cc:644] Failed to heartbeat to 127.6.234.254:36781 (0 consecutive failures): Service unavailable: Catalog manager is not initialized. State: Starting
I20260620 11:44:49.990229 20280 heartbeater.cc:464] Registering TS with master...
I20260620 11:44:49.990283 20280 heartbeater.cc:511] Master 127.6.234.254:36781 requested a full tablet report, sending...
I20260620 11:44:49.990605 20280 heartbeater.cc:464] Registering TS with master...
I20260620 11:44:49.990674 20280 heartbeater.cc:511] Master 127.6.234.254:36781 requested a full tablet report, sending...
I20260620 11:44:49.990685 20284 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 11:44:49.980169 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58618 (local address 127.6.234.254:36781)
0620 11:44:49.981796 (+ 1627us) server_negotiation.cc:206] Beginning negotiation
0620 11:44:49.981798 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 11:44:49.981805 (+ 7us) server_negotiation.cc:407] Connection header received
0620 11:44:49.981826 (+ 21us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 11:44:49.981827 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 11:44:49.981839 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 11:44:49.981902 (+ 63us) server_negotiation.cc:226] Negotiated authn=SASL
0620 11:44:49.983075 (+ 1173us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 11:44:49.983785 (+ 710us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 11:44:49.985151 (+ 1366us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 11:44:49.985750 (+ 599us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 11:44:49.985816 (+ 66us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 11:44:49.987265 (+ 1449us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 11:44:49.987266 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 11:44:49.987267 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 11:44:49.987274 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0620 11:44:49.989032 (+ 1758us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 11:44:49.989443 (+ 411us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 11:44:49.989444 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 11:44:49.989445 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 11:44:49.989505 (+ 60us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 11:44:49.989747 (+ 242us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 11:44:49.989748 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 11:44:49.989749 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 11:44:49.989879 (+ 130us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 11:44:49.990059 (+ 180us) server_negotiation.cc:1035] Waiting for connection context
0620 11:44:49.990511 (+ 452us) server_negotiation.cc:299] Negotiation successful
0620 11:44:49.990591 (+ 80us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1598,"thread_start_us":65,"threads_started":1}
W20260620 11:44:49.991207 20280 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
Jun 20 11:44:50 dist-test-slave-cjc6 krb5kdc[18818](info): AS_REQ (1 etypes {17}) 127.0.0.1: ISSUE: authtime 1781955890, etypes {rep=17 tkt=17 ses=17}, kudu/127.6.234.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 11:44:50.587992 20012 heartbeater.cc:464] Registering TS with master...
I20260620 11:44:50.588173 20012 heartbeater.cc:511] Master 127.6.234.254:36781 requested a full tablet report, sending...
I20260620 11:44:50.588698 19813 ts_manager.cc:194] Registered new tserver with Master: d6358116b2454b2fb815a367090682fd (127.6.234.193:38207)
I20260620 11:44:50.589743 19813 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.6.234.193@KRBTEST.COM'} at 127.6.234.193:46523
I20260620 11:44:50.790331 20146 heartbeater.cc:464] Registering TS with master...
I20260620 11:44:50.790508 20146 heartbeater.cc:511] Master 127.6.234.254:36781 requested a full tablet report, sending...
I20260620 11:44:50.790817 19813 ts_manager.cc:194] Registered new tserver with Master: c7d12946c3f1455fae82e6acc127fff2 (127.6.234.194:36505)
I20260620 11:44:50.791378 19813 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.6.234.194@KRBTEST.COM'} at 127.6.234.194:40117
Jun 20 11:44:50 dist-test-slave-cjc6 krb5kdc[18818](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781955890, etypes {rep=17 tkt=17 ses=17}, kudu/127.6.234.254@KRBTEST.COM for HTTP/127.6.234.212@KRBTEST.COM
I20260620 11:44:50.991395 20280 heartbeater.cc:464] Registering TS with master...
I20260620 11:44:50.991552 20280 heartbeater.cc:511] Master 127.6.234.254:36781 requested a full tablet report, sending...
I20260620 11:44:50.991938 19813 ts_manager.cc:194] Registered new tserver with Master: d7ee4c01f53e4ede9a24c5786ac0c0fa (127.6.234.195:41895)
I20260620 11:44:50.992468 19813 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.6.234.195@KRBTEST.COM'} at 127.6.234.195:60069
I20260620 11:44:50.995287 7083 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260620 11:44:51.004739 20301 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 11:44:50.996385 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58622 (local address 127.6.234.254:36781)
0620 11:44:50.996597 (+ 212us) server_negotiation.cc:206] Beginning negotiation
0620 11:44:50.996600 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0620 11:44:50.996724 (+ 124us) server_negotiation.cc:407] Connection header received
0620 11:44:50.996825 (+ 101us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 11:44:50.996826 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 11:44:50.996839 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 11:44:50.996884 (+ 45us) server_negotiation.cc:226] Negotiated authn=SASL
0620 11:44:50.997636 (+ 752us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 11:44:50.998433 (+ 797us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 11:44:50.999481 (+ 1048us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 11:44:50.999777 (+ 296us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 11:44:50.999824 (+ 47us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 11:44:51.001489 (+ 1665us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 11:44:51.001490 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 11:44:51.001490 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 11:44:51.001497 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0620 11:44:51.003386 (+ 1889us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 11:44:51.003818 (+ 432us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 11:44:51.003820 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 11:44:51.003820 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 11:44:51.003882 (+ 62us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 11:44:51.004084 (+ 202us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 11:44:51.004084 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 11:44:51.004085 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 11:44:51.004209 (+ 124us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 11:44:51.004285 (+ 76us) server_negotiation.cc:1035] Waiting for connection context
0620 11:44:51.004499 (+ 214us) server_negotiation.cc:299] Negotiation successful
0620 11:44:51.004614 (+ 115us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":189,"thread_start_us":68,"threads_started":1}
I20260620 11:44:51.007544 19813 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:58622:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260620 11:44:51.009819 19813 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260620 11:44:51.026719 20310 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 11:44:51.020106 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.234.195:41895 (local address 127.0.0.1:40612)
0620 11:44:51.021307 (+ 1201us) negotiation.cc:107] Waiting for socket to connect
0620 11:44:51.021348 (+ 41us) client_negotiation.cc:174] Beginning negotiation
0620 11:44:51.021418 (+ 70us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 11:44:51.021740 (+ 322us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 11:44:51.021743 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 11:44:51.021745 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 11:44:51.021843 (+ 98us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 11:44:51.021845 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 11:44:51.023594 (+ 1749us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 11:44:51.023595 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 11:44:51.024950 (+ 1355us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 11:44:51.024953 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 11:44:51.026407 (+ 1454us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 11:44:51.026408 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 11:44:51.026497 (+ 89us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 11:44:51.026513 (+ 16us) client_negotiation.cc:769] Sending connection context
0620 11:44:51.026579 (+ 66us) client_negotiation.cc:240] Negotiation successful
0620 11:44:51.026632 (+ 53us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1174,"thread_start_us":72,"threads_started":1}
I20260620 11:44:51.027175 20312 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 11:44:51.020914 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40612 (local address 127.6.234.195:41895)
0620 11:44:51.021242 (+ 328us) server_negotiation.cc:206] Beginning negotiation
0620 11:44:51.021248 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0620 11:44:51.021496 (+ 248us) server_negotiation.cc:407] Connection header received
0620 11:44:51.021533 (+ 37us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 11:44:51.021535 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 11:44:51.021551 (+ 16us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 11:44:51.021618 (+ 67us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 11:44:51.021930 (+ 312us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 11:44:51.023505 (+ 1575us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 11:44:51.025053 (+ 1548us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 11:44:51.026243 (+ 1190us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 11:44:51.026304 (+ 61us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 11:44:51.026368 (+ 64us) server_negotiation.cc:1035] Waiting for connection context
0620 11:44:51.027054 (+ 686us) server_negotiation.cc:299] Negotiation successful
0620 11:44:51.027097 (+ 43us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":294,"thread_start_us":65,"threads_started":1}
I20260620 11:44:51.028165 20196 tablet_service.cc:1467] Processing CreateTablet for tablet c20ab73145df4d1a97bb9a96f1ba4986 (DEFAULT_TABLE table=test-table [id=8a792d7e26104ec3b6fd1d89fac23e7b]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 11:44:51.028546 20196 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c20ab73145df4d1a97bb9a96f1ba4986. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 11:44:51.031602 20307 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 11:44:51.020361 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.234.194:36505 (local address 127.0.0.1:56954)
0620 11:44:51.023129 (+ 2768us) negotiation.cc:107] Waiting for socket to connect
0620 11:44:51.023140 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0620 11:44:51.023188 (+ 48us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 11:44:51.026326 (+ 3138us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 11:44:51.026327 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 11:44:51.026329 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 11:44:51.026382 (+ 53us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 11:44:51.026384 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 11:44:51.027621 (+ 1237us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 11:44:51.027622 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 11:44:51.028727 (+ 1105us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 11:44:51.028730 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 11:44:51.031347 (+ 2617us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 11:44:51.031348 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 11:44:51.031420 (+ 72us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 11:44:51.031425 (+ 5us) client_negotiation.cc:769] Sending connection context
0620 11:44:51.031478 (+ 53us) client_negotiation.cc:240] Negotiation successful
0620 11:44:51.031516 (+ 38us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":2748,"thread_start_us":65,"threads_started":1}
I20260620 11:44:51.038048 20308 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 11:44:51.020526 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.234.193:38207 (local address 127.0.0.1:41280)
0620 11:44:51.025832 (+ 5306us) negotiation.cc:107] Waiting for socket to connect
0620 11:44:51.025843 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0620 11:44:51.025924 (+ 81us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 11:44:51.033909 (+ 7985us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 11:44:51.033910 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 11:44:51.033913 (+ 3us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 11:44:51.033956 (+ 43us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 11:44:51.033958 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 11:44:51.035297 (+ 1339us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 11:44:51.035298 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 11:44:51.036391 (+ 1093us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 11:44:51.036394 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 11:44:51.037406 (+ 1012us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 11:44:51.037407 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 11:44:51.037871 (+ 464us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 11:44:51.037881 (+ 10us) client_negotiation.cc:769] Sending connection context
0620 11:44:51.037921 (+ 40us) client_negotiation.cc:240] Negotiation successful
0620 11:44:51.037965 (+ 44us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":5286,"thread_start_us":58,"threads_started":1}
I20260620 11:44:51.038648 20311 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 11:44:51.020833 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41280 (local address 127.6.234.193:38207)
0620 11:44:51.033717 (+ 12884us) server_negotiation.cc:206] Beginning negotiation
0620 11:44:51.033723 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0620 11:44:51.033734 (+ 11us) server_negotiation.cc:407] Connection header received
0620 11:44:51.033771 (+ 37us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 11:44:51.033773 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 11:44:51.033790 (+ 17us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 11:44:51.034045 (+ 255us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 11:44:51.034137 (+ 92us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 11:44:51.035192 (+ 1055us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 11:44:51.036499 (+ 1307us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 11:44:51.037286 (+ 787us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 11:44:51.038424 (+ 1138us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 11:44:51.038496 (+ 72us) server_negotiation.cc:1035] Waiting for connection context
0620 11:44:51.038524 (+ 28us) server_negotiation.cc:299] Negotiation successful
0620 11:44:51.038569 (+ 45us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":12823,"thread_start_us":52,"threads_started":1}
I20260620 11:44:51.039044 20313 tablet_bootstrap.cc:492] T c20ab73145df4d1a97bb9a96f1ba4986 P d7ee4c01f53e4ede9a24c5786ac0c0fa: Bootstrap starting.
I20260620 11:44:51.039557 19946 tablet_service.cc:1467] Processing CreateTablet for tablet c20ab73145df4d1a97bb9a96f1ba4986 (DEFAULT_TABLE table=test-table [id=8a792d7e26104ec3b6fd1d89fac23e7b]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 11:44:51.039849 20313 tablet_bootstrap.cc:654] T c20ab73145df4d1a97bb9a96f1ba4986 P d7ee4c01f53e4ede9a24c5786ac0c0fa: Neither blocks nor log segments found. Creating new log.
I20260620 11:44:51.039911 19946 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c20ab73145df4d1a97bb9a96f1ba4986. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 11:44:51.040174 20313 log.cc:824] T c20ab73145df4d1a97bb9a96f1ba4986 P d7ee4c01f53e4ede9a24c5786ac0c0fa: Log is configured to *not* fsync() on all Append() calls
I20260620 11:44:51.041033 20313 tablet_bootstrap.cc:492] T c20ab73145df4d1a97bb9a96f1ba4986 P d7ee4c01f53e4ede9a24c5786ac0c0fa: No bootstrap required, opened a new log
I20260620 11:44:51.041097 20313 ts_tablet_manager.cc:1397] T c20ab73145df4d1a97bb9a96f1ba4986 P d7ee4c01f53e4ede9a24c5786ac0c0fa: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260620 11:44:51.041957 20309 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 11:44:51.020477 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56954 (local address 127.6.234.194:36505)
0620 11:44:51.026043 (+ 5566us) server_negotiation.cc:206] Beginning negotiation
0620 11:44:51.026048 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 11:44:51.026059 (+ 11us) server_negotiation.cc:407] Connection header received
0620 11:44:51.026094 (+ 35us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 11:44:51.026096 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 11:44:51.026113 (+ 17us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 11:44:51.026189 (+ 76us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 11:44:51.026519 (+ 330us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 11:44:51.027523 (+ 1004us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 11:44:51.029684 (+ 2161us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 11:44:51.031153 (+ 1469us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 11:44:51.031237 (+ 84us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 11:44:51.031296 (+ 59us) server_negotiation.cc:1035] Waiting for connection context
0620 11:44:51.041774 (+ 10478us) server_negotiation.cc:299] Negotiation successful
0620 11:44:51.041853 (+ 79us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":5525,"thread_start_us":44,"threads_started":1}
I20260620 11:44:51.042609 20080 tablet_service.cc:1467] Processing CreateTablet for tablet c20ab73145df4d1a97bb9a96f1ba4986 (DEFAULT_TABLE table=test-table [id=8a792d7e26104ec3b6fd1d89fac23e7b]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 11:44:51.042927 20080 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c20ab73145df4d1a97bb9a96f1ba4986. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 11:44:51.043774 20313 raft_consensus.cc:348] T c20ab73145df4d1a97bb9a96f1ba4986 P d7ee4c01f53e4ede9a24c5786ac0c0fa [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d7ee4c01f53e4ede9a24c5786ac0c0fa" member_type: VOTER last_known_addr { host: "127.6.234.195" port: 41895 } } peers { permanent_uuid: "c7d12946c3f1455fae82e6acc127fff2" member_type: VOTER last_known_addr { host: "127.6.234.194" port: 36505 } } peers { permanent_uuid: "d6358116b2454b2fb815a367090682fd" member_type: VOTER last_known_addr { host: "127.6.234.193" port: 38207 } }
I20260620 11:44:51.044376 20313 raft_consensus.cc:374] T c20ab73145df4d1a97bb9a96f1ba4986 P d7ee4c01f53e4ede9a24c5786ac0c0fa [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 11:44:51.044482 20313 raft_consensus.cc:729] T c20ab73145df4d1a97bb9a96f1ba4986 P d7ee4c01f53e4ede9a24c5786ac0c0fa [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d7ee4c01f53e4ede9a24c5786ac0c0fa, State: Initialized, Role: FOLLOWER
I20260620 11:44:51.044682 20313 consensus_queue.cc:260] T c20ab73145df4d1a97bb9a96f1ba4986 P d7ee4c01f53e4ede9a24c5786ac0c0fa [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d7ee4c01f53e4ede9a24c5786ac0c0fa" member_type: VOTER last_known_addr { host: "127.6.234.195" port: 41895 } } peers { permanent_uuid: "c7d12946c3f1455fae82e6acc127fff2" member_type: VOTER last_known_addr { host: "127.6.234.194" port: 36505 } } peers { permanent_uuid: "d6358116b2454b2fb815a367090682fd" member_type: VOTER last_known_addr { host: "127.6.234.193" port: 38207 } }
I20260620 11:44:51.045081 20313 ts_tablet_manager.cc:1428] T c20ab73145df4d1a97bb9a96f1ba4986 P d7ee4c01f53e4ede9a24c5786ac0c0fa: Time spent starting tablet: real 0.004s user 0.004s sys 0.000s
I20260620 11:44:51.045239 20280 heartbeater.cc:503] Master 127.6.234.254:36781 was elected leader, sending a full tablet report...
I20260620 11:44:51.046243 20315 tablet_bootstrap.cc:492] T c20ab73145df4d1a97bb9a96f1ba4986 P d6358116b2454b2fb815a367090682fd: Bootstrap starting.
I20260620 11:44:51.047464 20315 tablet_bootstrap.cc:654] T c20ab73145df4d1a97bb9a96f1ba4986 P d6358116b2454b2fb815a367090682fd: Neither blocks nor log segments found. Creating new log.
I20260620 11:44:51.047937 20315 log.cc:824] T c20ab73145df4d1a97bb9a96f1ba4986 P d6358116b2454b2fb815a367090682fd: Log is configured to *not* fsync() on all Append() calls
I20260620 11:44:51.048319 20317 tablet_bootstrap.cc:492] T c20ab73145df4d1a97bb9a96f1ba4986 P c7d12946c3f1455fae82e6acc127fff2: Bootstrap starting.
I20260620 11:44:51.049002 20317 tablet_bootstrap.cc:654] T c20ab73145df4d1a97bb9a96f1ba4986 P c7d12946c3f1455fae82e6acc127fff2: Neither blocks nor log segments found. Creating new log.
I20260620 11:44:51.049373 20317 log.cc:824] T c20ab73145df4d1a97bb9a96f1ba4986 P c7d12946c3f1455fae82e6acc127fff2: Log is configured to *not* fsync() on all Append() calls
I20260620 11:44:51.050246 20315 tablet_bootstrap.cc:492] T c20ab73145df4d1a97bb9a96f1ba4986 P d6358116b2454b2fb815a367090682fd: No bootstrap required, opened a new log
I20260620 11:44:51.050711 20315 ts_tablet_manager.cc:1397] T c20ab73145df4d1a97bb9a96f1ba4986 P d6358116b2454b2fb815a367090682fd: Time spent bootstrapping tablet: real 0.005s user 0.002s sys 0.000s
I20260620 11:44:51.053478 20315 raft_consensus.cc:348] T c20ab73145df4d1a97bb9a96f1ba4986 P d6358116b2454b2fb815a367090682fd [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d7ee4c01f53e4ede9a24c5786ac0c0fa" member_type: VOTER last_known_addr { host: "127.6.234.195" port: 41895 } } peers { permanent_uuid: "c7d12946c3f1455fae82e6acc127fff2" member_type: VOTER last_known_addr { host: "127.6.234.194" port: 36505 } } peers { permanent_uuid: "d6358116b2454b2fb815a367090682fd" member_type: VOTER last_known_addr { host: "127.6.234.193" port: 38207 } }
I20260620 11:44:51.053767 20315 raft_consensus.cc:374] T c20ab73145df4d1a97bb9a96f1ba4986 P d6358116b2454b2fb815a367090682fd [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 11:44:51.053911 20315 raft_consensus.cc:729] T c20ab73145df4d1a97bb9a96f1ba4986 P d6358116b2454b2fb815a367090682fd [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d6358116b2454b2fb815a367090682fd, State: Initialized, Role: FOLLOWER
I20260620 11:44:51.054137 20315 consensus_queue.cc:260] T c20ab73145df4d1a97bb9a96f1ba4986 P d6358116b2454b2fb815a367090682fd [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d7ee4c01f53e4ede9a24c5786ac0c0fa" member_type: VOTER last_known_addr { host: "127.6.234.195" port: 41895 } } peers { permanent_uuid: "c7d12946c3f1455fae82e6acc127fff2" member_type: VOTER last_known_addr { host: "127.6.234.194" port: 36505 } } peers { permanent_uuid: "d6358116b2454b2fb815a367090682fd" member_type: VOTER last_known_addr { host: "127.6.234.193" port: 38207 } }
I20260620 11:44:51.054471 20315 ts_tablet_manager.cc:1428] T c20ab73145df4d1a97bb9a96f1ba4986 P d6358116b2454b2fb815a367090682fd: Time spent starting tablet: real 0.004s user 0.003s sys 0.000s
I20260620 11:44:51.054629 20012 heartbeater.cc:503] Master 127.6.234.254:36781 was elected leader, sending a full tablet report...
I20260620 11:44:51.055954 20317 tablet_bootstrap.cc:492] T c20ab73145df4d1a97bb9a96f1ba4986 P c7d12946c3f1455fae82e6acc127fff2: No bootstrap required, opened a new log
I20260620 11:44:51.056078 20317 ts_tablet_manager.cc:1397] T c20ab73145df4d1a97bb9a96f1ba4986 P c7d12946c3f1455fae82e6acc127fff2: Time spent bootstrapping tablet: real 0.008s user 0.002s sys 0.000s
I20260620 11:44:51.059119 20317 raft_consensus.cc:348] T c20ab73145df4d1a97bb9a96f1ba4986 P c7d12946c3f1455fae82e6acc127fff2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d7ee4c01f53e4ede9a24c5786ac0c0fa" member_type: VOTER last_known_addr { host: "127.6.234.195" port: 41895 } } peers { permanent_uuid: "c7d12946c3f1455fae82e6acc127fff2" member_type: VOTER last_known_addr { host: "127.6.234.194" port: 36505 } } peers { permanent_uuid: "d6358116b2454b2fb815a367090682fd" member_type: VOTER last_known_addr { host: "127.6.234.193" port: 38207 } }
I20260620 11:44:51.059342 20317 raft_consensus.cc:374] T c20ab73145df4d1a97bb9a96f1ba4986 P c7d12946c3f1455fae82e6acc127fff2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 11:44:51.059419 20317 raft_consensus.cc:729] T c20ab73145df4d1a97bb9a96f1ba4986 P c7d12946c3f1455fae82e6acc127fff2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c7d12946c3f1455fae82e6acc127fff2, State: Initialized, Role: FOLLOWER
I20260620 11:44:51.059595 20317 consensus_queue.cc:260] T c20ab73145df4d1a97bb9a96f1ba4986 P c7d12946c3f1455fae82e6acc127fff2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d7ee4c01f53e4ede9a24c5786ac0c0fa" member_type: VOTER last_known_addr { host: "127.6.234.195" port: 41895 } } peers { permanent_uuid: "c7d12946c3f1455fae82e6acc127fff2" member_type: VOTER last_known_addr { host: "127.6.234.194" port: 36505 } } peers { permanent_uuid: "d6358116b2454b2fb815a367090682fd" member_type: VOTER last_known_addr { host: "127.6.234.193" port: 38207 } }
I20260620 11:44:51.059924 20317 ts_tablet_manager.cc:1428] T c20ab73145df4d1a97bb9a96f1ba4986 P c7d12946c3f1455fae82e6acc127fff2: Time spent starting tablet: real 0.004s user 0.003s sys 0.000s
I20260620 11:44:51.060075 20146 heartbeater.cc:503] Master 127.6.234.254:36781 was elected leader, sending a full tablet report...
I20260620 11:44:51.071873 20316 raft_consensus.cc:482] T c20ab73145df4d1a97bb9a96f1ba4986 P d7ee4c01f53e4ede9a24c5786ac0c0fa [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260620 11:44:51.072021 20316 raft_consensus.cc:504] T c20ab73145df4d1a97bb9a96f1ba4986 P d7ee4c01f53e4ede9a24c5786ac0c0fa [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d7ee4c01f53e4ede9a24c5786ac0c0fa" member_type: VOTER last_known_addr { host: "127.6.234.195" port: 41895 } } peers { permanent_uuid: "c7d12946c3f1455fae82e6acc127fff2" member_type: VOTER last_known_addr { host: "127.6.234.194" port: 36505 } } peers { permanent_uuid: "d6358116b2454b2fb815a367090682fd" member_type: VOTER last_known_addr { host: "127.6.234.193" port: 38207 } }
I20260620 11:44:51.072422 20316 leader_election.cc:288] T c20ab73145df4d1a97bb9a96f1ba4986 P d7ee4c01f53e4ede9a24c5786ac0c0fa [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers c7d12946c3f1455fae82e6acc127fff2 (127.6.234.194:36505), d6358116b2454b2fb815a367090682fd (127.6.234.193:38207)
W20260620 11:44:51.072575 20013 tablet.cc:2369] T c20ab73145df4d1a97bb9a96f1ba4986 P d6358116b2454b2fb815a367090682fd: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260620 11:44:51.079696 20311 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 11:44:51.073296 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.6.234.195:42623 (local address 127.6.234.193:38207)
0620 11:44:51.073339 (+ 43us) server_negotiation.cc:206] Beginning negotiation
0620 11:44:51.073341 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 11:44:51.076102 (+ 2761us) server_negotiation.cc:407] Connection header received
0620 11:44:51.076198 (+ 96us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 11:44:51.076200 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 11:44:51.076209 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 11:44:51.076239 (+ 30us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 11:44:51.076448 (+ 209us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 11:44:51.077177 (+ 729us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 11:44:51.078725 (+ 1548us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 11:44:51.079262 (+ 537us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 11:44:51.079317 (+ 55us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 11:44:51.079327 (+ 10us) server_negotiation.cc:1035] Waiting for connection context
0620 11:44:51.079577 (+ 250us) server_negotiation.cc:299] Negotiation successful
0620 11:44:51.079622 (+ 45us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":21}
I20260620 11:44:51.080157 20322 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 11:44:51.072985 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.234.193:38207 (local address 127.6.234.195:42623)
0620 11:44:51.076046 (+ 3061us) negotiation.cc:107] Waiting for socket to connect
0620 11:44:51.076055 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0620 11:44:51.076126 (+ 71us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 11:44:51.076302 (+ 176us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 11:44:51.076302 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 11:44:51.076304 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 11:44:51.076352 (+ 48us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 11:44:51.076353 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 11:44:51.077276 (+ 923us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 11:44:51.077277 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 11:44:51.078622 (+ 1345us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 11:44:51.078625 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 11:44:51.079390 (+ 765us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 11:44:51.079391 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 11:44:51.079479 (+ 88us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 11:44:51.079484 (+ 5us) client_negotiation.cc:769] Sending connection context
0620 11:44:51.080015 (+ 531us) client_negotiation.cc:240] Negotiation successful
0620 11:44:51.080061 (+ 46us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":3044,"thread_start_us":66,"threads_started":1}
I20260620 11:44:51.080647 19964 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c20ab73145df4d1a97bb9a96f1ba4986" candidate_uuid: "d7ee4c01f53e4ede9a24c5786ac0c0fa" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d6358116b2454b2fb815a367090682fd" is_pre_election: true
I20260620 11:44:51.080811 19964 raft_consensus.cc:2454] T c20ab73145df4d1a97bb9a96f1ba4986 P d6358116b2454b2fb815a367090682fd [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d7ee4c01f53e4ede9a24c5786ac0c0fa in term 0.
I20260620 11:44:51.081143 20167 leader_election.cc:302] T c20ab73145df4d1a97bb9a96f1ba4986 P d7ee4c01f53e4ede9a24c5786ac0c0fa [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: d6358116b2454b2fb815a367090682fd, d7ee4c01f53e4ede9a24c5786ac0c0fa; no voters:
I20260620 11:44:51.081265 20316 raft_consensus.cc:2781] T c20ab73145df4d1a97bb9a96f1ba4986 P d7ee4c01f53e4ede9a24c5786ac0c0fa [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260620 11:44:51.081301 20316 raft_consensus.cc:482] T c20ab73145df4d1a97bb9a96f1ba4986 P d7ee4c01f53e4ede9a24c5786ac0c0fa [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260620 11:44:51.081322 20316 raft_consensus.cc:3037] T c20ab73145df4d1a97bb9a96f1ba4986 P d7ee4c01f53e4ede9a24c5786ac0c0fa [term 0 FOLLOWER]: Advancing to term 1
I20260620 11:44:51.082242 20316 raft_consensus.cc:504] T c20ab73145df4d1a97bb9a96f1ba4986 P d7ee4c01f53e4ede9a24c5786ac0c0fa [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d7ee4c01f53e4ede9a24c5786ac0c0fa" member_type: VOTER last_known_addr { host: "127.6.234.195" port: 41895 } } peers { permanent_uuid: "c7d12946c3f1455fae82e6acc127fff2" member_type: VOTER last_known_addr { host: "127.6.234.194" port: 36505 } } peers { permanent_uuid: "d6358116b2454b2fb815a367090682fd" member_type: VOTER last_known_addr { host: "127.6.234.193" port: 38207 } }
I20260620 11:44:51.082367 20316 leader_election.cc:288] T c20ab73145df4d1a97bb9a96f1ba4986 P d7ee4c01f53e4ede9a24c5786ac0c0fa [CANDIDATE]: Term 1 election: Requested vote from peers c7d12946c3f1455fae82e6acc127fff2 (127.6.234.194:36505), d6358116b2454b2fb815a367090682fd (127.6.234.193:38207)
I20260620 11:44:51.082667 19964 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c20ab73145df4d1a97bb9a96f1ba4986" candidate_uuid: "d7ee4c01f53e4ede9a24c5786ac0c0fa" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d6358116b2454b2fb815a367090682fd"
I20260620 11:44:51.082737 19964 raft_consensus.cc:3037] T c20ab73145df4d1a97bb9a96f1ba4986 P d6358116b2454b2fb815a367090682fd [term 0 FOLLOWER]: Advancing to term 1
I20260620 11:44:51.083503 19964 raft_consensus.cc:2454] T c20ab73145df4d1a97bb9a96f1ba4986 P d6358116b2454b2fb815a367090682fd [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d7ee4c01f53e4ede9a24c5786ac0c0fa in term 1.
I20260620 11:44:51.083709 20167 leader_election.cc:302] T c20ab73145df4d1a97bb9a96f1ba4986 P d7ee4c01f53e4ede9a24c5786ac0c0fa [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: d6358116b2454b2fb815a367090682fd, d7ee4c01f53e4ede9a24c5786ac0c0fa; no voters:
I20260620 11:44:51.083823 20316 raft_consensus.cc:2781] T c20ab73145df4d1a97bb9a96f1ba4986 P d7ee4c01f53e4ede9a24c5786ac0c0fa [term 1 FOLLOWER]: Leader election won for term 1
I20260620 11:44:51.083968 20316 raft_consensus.cc:686] T c20ab73145df4d1a97bb9a96f1ba4986 P d7ee4c01f53e4ede9a24c5786ac0c0fa [term 1 LEADER]: Becoming Leader. State: Replica: d7ee4c01f53e4ede9a24c5786ac0c0fa, State: Running, Role: LEADER
I20260620 11:44:51.084049 20316 consensus_queue.cc:237] T c20ab73145df4d1a97bb9a96f1ba4986 P d7ee4c01f53e4ede9a24c5786ac0c0fa [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d7ee4c01f53e4ede9a24c5786ac0c0fa" member_type: VOTER last_known_addr { host: "127.6.234.195" port: 41895 } } peers { permanent_uuid: "c7d12946c3f1455fae82e6acc127fff2" member_type: VOTER last_known_addr { host: "127.6.234.194" port: 36505 } } peers { permanent_uuid: "d6358116b2454b2fb815a367090682fd" member_type: VOTER last_known_addr { host: "127.6.234.193" port: 38207 } }
I20260620 11:44:51.084553 20309 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 11:44:51.073505 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.6.234.195:55553 (local address 127.6.234.194:36505)
0620 11:44:51.073550 (+ 45us) server_negotiation.cc:206] Beginning negotiation
0620 11:44:51.073551 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0620 11:44:51.075920 (+ 2369us) server_negotiation.cc:407] Connection header received
0620 11:44:51.076005 (+ 85us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 11:44:51.076006 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 11:44:51.076017 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 11:44:51.076076 (+ 59us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 11:44:51.081572 (+ 5496us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 11:44:51.082287 (+ 715us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 11:44:51.083600 (+ 1313us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 11:44:51.084174 (+ 574us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 11:44:51.084223 (+ 49us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 11:44:51.084232 (+ 9us) server_negotiation.cc:1035] Waiting for connection context
0620 11:44:51.084448 (+ 216us) server_negotiation.cc:299] Negotiation successful
0620 11:44:51.084491 (+ 43us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":23}
I20260620 11:44:51.084748 20323 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 11:44:51.072765 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.234.194:36505 (local address 127.6.234.195:55553)
0620 11:44:51.075845 (+ 3080us) negotiation.cc:107] Waiting for socket to connect
0620 11:44:51.075859 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0620 11:44:51.075937 (+ 78us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 11:44:51.076129 (+ 192us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 11:44:51.076130 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 11:44:51.076132 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 11:44:51.076215 (+ 83us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 11:44:51.076217 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 11:44:51.082390 (+ 6173us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 11:44:51.082390 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 11:44:51.083488 (+ 1098us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 11:44:51.083491 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 11:44:51.084287 (+ 796us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 11:44:51.084288 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 11:44:51.084359 (+ 71us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 11:44:51.084364 (+ 5us) client_negotiation.cc:769] Sending connection context
0620 11:44:51.084619 (+ 255us) client_negotiation.cc:240] Negotiation successful
0620 11:44:51.084664 (+ 45us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":3058,"thread_start_us":103,"threads_started":1}
I20260620 11:44:51.084897 19813 catalog_manager.cc:5495] T c20ab73145df4d1a97bb9a96f1ba4986 P d7ee4c01f53e4ede9a24c5786ac0c0fa reported cstate change: term changed from 0 to 1, leader changed from <none> to d7ee4c01f53e4ede9a24c5786ac0c0fa (127.6.234.195). New cstate: current_term: 1 leader_uuid: "d7ee4c01f53e4ede9a24c5786ac0c0fa" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d7ee4c01f53e4ede9a24c5786ac0c0fa" member_type: VOTER last_known_addr { host: "127.6.234.195" port: 41895 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "c7d12946c3f1455fae82e6acc127fff2" member_type: VOTER last_known_addr { host: "127.6.234.194" port: 36505 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "d6358116b2454b2fb815a367090682fd" member_type: VOTER last_known_addr { host: "127.6.234.193" port: 38207 } health_report { overall_health: UNKNOWN } } }
I20260620 11:44:51.086638 20100 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c20ab73145df4d1a97bb9a96f1ba4986" candidate_uuid: "d7ee4c01f53e4ede9a24c5786ac0c0fa" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c7d12946c3f1455fae82e6acc127fff2" is_pre_election: true
I20260620 11:44:51.086799 20100 raft_consensus.cc:2454] T c20ab73145df4d1a97bb9a96f1ba4986 P c7d12946c3f1455fae82e6acc127fff2 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d7ee4c01f53e4ede9a24c5786ac0c0fa in term 0.
I20260620 11:44:51.086935 20099 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c20ab73145df4d1a97bb9a96f1ba4986" candidate_uuid: "d7ee4c01f53e4ede9a24c5786ac0c0fa" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c7d12946c3f1455fae82e6acc127fff2"
I20260620 11:44:51.086988 20099 raft_consensus.cc:3037] T c20ab73145df4d1a97bb9a96f1ba4986 P c7d12946c3f1455fae82e6acc127fff2 [term 0 FOLLOWER]: Advancing to term 1
I20260620 11:44:51.087847 20099 raft_consensus.cc:2454] T c20ab73145df4d1a97bb9a96f1ba4986 P c7d12946c3f1455fae82e6acc127fff2 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d7ee4c01f53e4ede9a24c5786ac0c0fa in term 1.
I20260620 11:44:51.111584 20312 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 11:44:51.108158 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40622 (local address 127.6.234.195:41895)
0620 11:44:51.108213 (+ 55us) server_negotiation.cc:206] Beginning negotiation
0620 11:44:51.108215 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 11:44:51.108222 (+ 7us) server_negotiation.cc:407] Connection header received
0620 11:44:51.108261 (+ 39us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 11:44:51.108262 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 11:44:51.108283 (+ 21us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 11:44:51.108362 (+ 79us) server_negotiation.cc:226] Negotiated authn=TOKEN
0620 11:44:51.108565 (+ 203us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 11:44:51.109306 (+ 741us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 11:44:51.110480 (+ 1174us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 11:44:51.110774 (+ 294us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 11:44:51.110837 (+ 63us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 11:44:51.111034 (+ 197us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0620 11:44:51.111172 (+ 138us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0620 11:44:51.111257 (+ 85us) server_negotiation.cc:1035] Waiting for connection context
0620 11:44:51.111481 (+ 224us) server_negotiation.cc:299] Negotiation successful
0620 11:44:51.111504 (+ 23us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":25}
I20260620 11:44:51.118870 19964 raft_consensus.cc:1261] T c20ab73145df4d1a97bb9a96f1ba4986 P d6358116b2454b2fb815a367090682fd [term 1 FOLLOWER]: Refusing update from remote peer d7ee4c01f53e4ede9a24c5786ac0c0fa: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 11:44:51.119244 20324 consensus_queue.cc:1035] T c20ab73145df4d1a97bb9a96f1ba4986 P d7ee4c01f53e4ede9a24c5786ac0c0fa [LEADER]: Connected to new peer: Peer: permanent_uuid: "d6358116b2454b2fb815a367090682fd" member_type: VOTER last_known_addr { host: "127.6.234.193" port: 38207 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 11:44:51.121347 20099 raft_consensus.cc:1261] T c20ab73145df4d1a97bb9a96f1ba4986 P c7d12946c3f1455fae82e6acc127fff2 [term 1 FOLLOWER]: Refusing update from remote peer d7ee4c01f53e4ede9a24c5786ac0c0fa: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 11:44:51.121587 20324 consensus_queue.cc:1035] T c20ab73145df4d1a97bb9a96f1ba4986 P d7ee4c01f53e4ede9a24c5786ac0c0fa [LEADER]: Connected to new peer: Peer: permanent_uuid: "c7d12946c3f1455fae82e6acc127fff2" member_type: VOTER last_known_addr { host: "127.6.234.194" port: 36505 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 11:44:51.129768 20328 mvcc.cc:205] Tried to move back new op lower bound from 7298891330019700736 to 7298891329880821760. Current Snapshot: MvccSnapshot[applied={T|T < 7298891330019700736}]
I20260620 11:44:51.130142 20327 mvcc.cc:205] Tried to move back new op lower bound from 7298891330019700736 to 7298891329880821760. Current Snapshot: MvccSnapshot[applied={T|T < 7298891330019700736}]
I20260620 11:44:51.132112 20330 mvcc.cc:205] Tried to move back new op lower bound from 7298891330019700736 to 7298891329880821760. Current Snapshot: MvccSnapshot[applied={T|T < 7298891330019700736}]
I20260620 11:44:51.136010 19813 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:58622:
table { table_name: "test-table" } modify_external_catalogs: true
I20260620 11:44:51.136114 19813 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:58622:
table { table_name: "test-table" } modify_external_catalogs: true
I20260620 11:44:51.136828 19813 catalog_manager.cc:5782] T 00000000000000000000000000000000 P cd8640322bce49ccb95296ca8860fadd: Sending DeleteTablet for 3 replicas of tablet c20ab73145df4d1a97bb9a96f1ba4986
I20260620 11:44:51.137251 20196 tablet_service.cc:1514] Processing DeleteTablet for tablet c20ab73145df4d1a97bb9a96f1ba4986 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 11:44:51 UTC) from {username='kudu', principal='kudu/127.6.234.254@KRBTEST.COM'} at 127.0.0.1:40612
I20260620 11:44:51.137463 20080 tablet_service.cc:1514] Processing DeleteTablet for tablet c20ab73145df4d1a97bb9a96f1ba4986 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 11:44:51 UTC) from {username='kudu', principal='kudu/127.6.234.254@KRBTEST.COM'} at 127.0.0.1:56954
I20260620 11:44:51.137749 19946 tablet_service.cc:1514] Processing DeleteTablet for tablet c20ab73145df4d1a97bb9a96f1ba4986 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 11:44:51 UTC) from {username='kudu', principal='kudu/127.6.234.254@KRBTEST.COM'} at 127.0.0.1:41280
I20260620 11:44:51.138147 7083 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task1JRg7N/build/release/bin/kudu with pid 19852
I20260620 11:44:51.146128 20336 tablet_replica.cc:330] T c20ab73145df4d1a97bb9a96f1ba4986 P d7ee4c01f53e4ede9a24c5786ac0c0fa: stopping tablet replica
I20260620 11:44:51.146255 20336 raft_consensus.cc:2229] T c20ab73145df4d1a97bb9a96f1ba4986 P d7ee4c01f53e4ede9a24c5786ac0c0fa [term 1 LEADER]: Raft consensus shutting down.
I20260620 11:44:51.146407 20336 raft_consensus.cc:2258] T c20ab73145df4d1a97bb9a96f1ba4986 P d7ee4c01f53e4ede9a24c5786ac0c0fa [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 11:44:51.147214 20337 tablet_replica.cc:330] T c20ab73145df4d1a97bb9a96f1ba4986 P c7d12946c3f1455fae82e6acc127fff2: stopping tablet replica
I20260620 11:44:51.147334 20337 raft_consensus.cc:2229] T c20ab73145df4d1a97bb9a96f1ba4986 P c7d12946c3f1455fae82e6acc127fff2 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260620 11:44:51.147439 20337 raft_consensus.cc:2258] T c20ab73145df4d1a97bb9a96f1ba4986 P c7d12946c3f1455fae82e6acc127fff2 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 11:44:51.147990 20337 ts_tablet_manager.cc:1905] T c20ab73145df4d1a97bb9a96f1ba4986 P c7d12946c3f1455fae82e6acc127fff2: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 11:44:51.148535 20336 ts_tablet_manager.cc:1905] T c20ab73145df4d1a97bb9a96f1ba4986 P d7ee4c01f53e4ede9a24c5786ac0c0fa: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 11:44:51.152047 20337 ts_tablet_manager.cc:1918] T c20ab73145df4d1a97bb9a96f1ba4986 P c7d12946c3f1455fae82e6acc127fff2: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260620 11:44:51.152127 20337 log.cc:1196] T c20ab73145df4d1a97bb9a96f1ba4986 P c7d12946c3f1455fae82e6acc127fff2: Deleting WAL directory at /tmp/dist-test-task1JRg7N/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781955746769051-7083-0/minicluster-data/ts-1/wal/wals/c20ab73145df4d1a97bb9a96f1ba4986
I20260620 11:44:51.152398 20337 ts_tablet_manager.cc:1939] T c20ab73145df4d1a97bb9a96f1ba4986 P c7d12946c3f1455fae82e6acc127fff2: Deleting consensus metadata
I20260620 11:44:51.161846 20336 ts_tablet_manager.cc:1918] T c20ab73145df4d1a97bb9a96f1ba4986 P d7ee4c01f53e4ede9a24c5786ac0c0fa: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260620 11:44:51.161954 20336 log.cc:1196] T c20ab73145df4d1a97bb9a96f1ba4986 P d7ee4c01f53e4ede9a24c5786ac0c0fa: Deleting WAL directory at /tmp/dist-test-task1JRg7N/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781955746769051-7083-0/minicluster-data/ts-2/wal/wals/c20ab73145df4d1a97bb9a96f1ba4986
I20260620 11:44:51.162261 20336 ts_tablet_manager.cc:1939] T c20ab73145df4d1a97bb9a96f1ba4986 P d7ee4c01f53e4ede9a24c5786ac0c0fa: Deleting consensus metadata
I20260620 11:44:51.162736 19799 catalog_manager.cc:4841] TS d7ee4c01f53e4ede9a24c5786ac0c0fa (127.6.234.195:41895): tablet c20ab73145df4d1a97bb9a96f1ba4986 (table test-table [id=8a792d7e26104ec3b6fd1d89fac23e7b]) successfully deleted
I20260620 11:44:51.163208 19798 catalog_manager.cc:4841] TS c7d12946c3f1455fae82e6acc127fff2 (127.6.234.194:36505): tablet c20ab73145df4d1a97bb9a96f1ba4986 (table test-table [id=8a792d7e26104ec3b6fd1d89fac23e7b]) successfully deleted
W20260620 11:44:51.163630 19798 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.6.234.193:38207 (error 108)
I20260620 11:44:51.163949 7083 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task1JRg7N/build/release/bin/kudu with pid 20016
I20260620 11:44:51.178607 20308 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 11:44:51.178444 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.234.193:38207 (local address 127.0.0.1:41286)
0620 11:44:51.178499 (+ 55us) negotiation.cc:107] Waiting for socket to connect
0620 11:44:51.178558 (+ 59us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.234.193:38207: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":26}
I20260620 11:44:51.178602 7083 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task1JRg7N/build/release/bin/kudu with pid 20149
W20260620 11:44:51.181030 19798 catalog_manager.cc:4584] TS d6358116b2454b2fb815a367090682fd (127.6.234.193:38207): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet c20ab73145df4d1a97bb9a96f1ba4986: Network error: Client connection negotiation failed: client connection to 127.6.234.193:38207: connect: Connection refused (error 111)
Jun 20 11:44:51 dist-test-slave-cjc6 krb5kdc[18818](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781955890, etypes {rep=17 tkt=17 ses=17}, kudu/127.6.234.254@KRBTEST.COM for HTTP/127.6.234.212@KRBTEST.COM
I20260620 11:44:51.184873 7083 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task1JRg7N/build/release/bin/kudu with pid 19781
I20260620 11:44:51.191879 7083 mini_ranger_kms.cc:60] Stopping Ranger KMS...
I20260620 11:44:51.536592 7083 mini_ranger_kms.cc:62] Stopped Ranger KMS
I20260620 11:44:51.536688 7083 mini_ranger.cc:66] Stopping Ranger...
I20260620 11:44:51.881054 7083 mini_ranger.cc:68] Stopped Ranger
2026-06-20 11:44:51.881 UTC [18848] LOG: received smart shutdown request
2026-06-20 11:44:51.885 UTC [18848] LOG: background worker "logical replication launcher" (PID 18857) exited with exit code 1
2026-06-20 11:44:51.885 UTC [18852] LOG: shutting down
2026-06-20 11:44:52.047 UTC [18848] LOG: database system is shut down
2026-06-20T11:44:52Z chronyd exiting
[ OK ] SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers (46385 ms)
[ RUN ] SecurityITest.IPKICACert
Loading random data
Initializing database '/tmp/dist-test-task1JRg7N/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781955746769051-7083-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 20 11:44:52 dist-test-slave-cjc6 krb5kdc[20349](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 20 11:44:52 dist-test-slave-cjc6 krb5kdc[20349](info): set up 2 sockets
Jun 20 11:44:52 dist-test-slave-cjc6 krb5kdc[20349](info): commencing operation
krb5kdc: starting...
W20260620 11:44:53.197767 7083 mini_kdc.cc:121] Time spent starting KDC: real 1.022s user 0.000s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task1JRg7N/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781955746769051-7083-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task1JRg7N/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781955746769051-7083-0/krb5kdc/test-user.keytab.
Jun 20 11:44:53 dist-test-slave-cjc6 krb5kdc[20349](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781955893, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-20T11:44:53Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-20T11:44:53Z Disabled control of system clock
WARNING: no policy specified for kudu/127.6.234.254@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.6.234.254@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.6.234.254 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task1JRg7N/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781955746769051-7083-0/krb5kdc/kudu_127.6.234.254.keytab.
Entry for principal kudu/127.6.234.254 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task1JRg7N/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781955746769051-7083-0/krb5kdc/kudu_127.6.234.254.keytab.
I20260620 11:44:53.316498 7083 external_mini_cluster.cc:1326] Running /tmp/dist-test-task1JRg7N/build/release/bin/kudu
/tmp/dist-test-task1JRg7N/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task1JRg7N/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781955746769051-7083-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task1JRg7N/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781955746769051-7083-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task1JRg7N/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781955746769051-7083-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task1JRg7N/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781955746769051-7083-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.6.234.254:35005
--webserver_interface=127.6.234.254
--webserver_port=0
--keytab_file=/tmp/dist-test-task1JRg7N/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781955746769051-7083-0/krb5kdc/kudu_127.6.234.254.keytab
--principal=kudu/127.6.234.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.6.234.212:44363
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.6.234.254:35005,127.6.234.253:41267,127.6.234.252:38097
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task1JRg7N/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781955746769051-7083-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task1JRg7N/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781955746769051-7083-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task1JRg7N/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781955746769051-7083-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 11:44:53.395527 20362 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 11:44:53.395725 20362 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 11:44:53.395747 20362 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 11:44:53.396799 20362 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 11:44:53.396831 20362 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 11:44:53.396845 20362 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 11:44:53.396857 20362 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 11:44:53.396869 20362 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 11:44:53.398348 20362 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.6.234.212:44363
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task1JRg7N/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781955746769051-7083-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task1JRg7N/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781955746769051-7083-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.6.234.254:35005,127.6.234.253:41267,127.6.234.252:38097
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task1JRg7N/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781955746769051-7083-0/krb5kdc/kudu_127.6.234.254.keytab
--principal=kudu/127.6.234.254
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.6.234.254:35005
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task1JRg7N/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781955746769051-7083-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.6.234.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.20362
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task1JRg7N/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781955746769051-7083-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 20 Jun 2026 10:32:37 UTC on ccycloud.quasar-bgyofv.root.comops.site
build id 4457173
I20260620 11:44:53.398535 20362 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 11:44:53.398694 20362 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 11:44:53.403784 20367 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 11:44:53.403793 20370 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 11:44:53.403793 20368 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 11:44:53.403919 20362 server_base.cc:1034] running on GCE node
I20260620 11:44:53.404235 20362 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 11:44:53.404536 20362 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 11:44:53.405753 20362 hybrid_clock.cc:648] HybridClock initialized: now 1781955893405756 us; error 47 us; skew 500 ppm
Jun 20 11:44:53 dist-test-slave-cjc6 krb5kdc[20349](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781955893, etypes {rep=17 tkt=17 ses=17}, kudu/127.6.234.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 11:44:53.412402 20362 init.cc:377] Logged in from keytab as kudu/127.6.234.254@KRBTEST.COM (short username kudu)
I20260620 11:44:53.412909 20362 webserver.cc:460] Webserver started at http://127.6.234.254:37785/ using document root <none> and password file <none>
I20260620 11:44:53.413137 20362 fs_manager.cc:359] Metadata directory not provided
I20260620 11:44:53.413213 20362 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 11:44:53.413370 20362 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 11:44:53.414484 20362 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task1JRg7N/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781955746769051-7083-0/minicluster-data/master-0/data/instance:
uuid: "619c5ff12b6949eea40c74e81640db02"
format_stamp: "Formatted at 2026-06-20 11:44:53 on dist-test-slave-cjc6"
server_key: "33657f1f64b1de3c4dd1f5575de6e585"
server_key_iv: "d8bd84e108734362f662544b91c399e4"
server_key_version: "encryptionkey@0"
I20260620 11:44:53.414850 20362 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task1JRg7N/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781955746769051-7083-0/minicluster-data/master-0/wal/instance:
uuid: "619c5ff12b6949eea40c74e81640db02"
format_stamp: "Formatted at 2026-06-20 11:44:53 on dist-test-slave-cjc6"
server_key: "33657f1f64b1de3c4dd1f5575de6e585"
server_key_iv: "d8bd84e108734362f662544b91c399e4"
server_key_version: "encryptionkey@0"
I20260620 11:44:53.416160 20362 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.002s
I20260620 11:44:53.416882 20377 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 11:44:53.417057 20362 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260620 11:44:53.417166 20362 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task1JRg7N/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781955746769051-7083-0/minicluster-data/master-0/data,/tmp/dist-test-task1JRg7N/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781955746769051-7083-0/minicluster-data/master-0/wal
uuid: "619c5ff12b6949eea40c74e81640db02"
format_stamp: "Formatted at 2026-06-20 11:44:53 on dist-test-slave-cjc6"
server_key: "33657f1f64b1de3c4dd1f5575de6e585"
server_key_iv: "d8bd84e108734362f662544b91c399e4"
server_key_version: "encryptionkey@0"
I20260620 11:44:53.417250 20362 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task1JRg7N/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781955746769051-7083-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task1JRg7N/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781955746769051-7083-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task1JRg7N/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781955746769051-7083-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 11:44:53.432345 20362 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 11:44:53.434384 20362 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 11:44:53.434551 20362 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 11:44:53.438746 20362 rpc_server.cc:307] RPC server started. Bound to: 127.6.234.254:35005
I20260620 11:44:53.438782 20429 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.6.234.254:35005 every 8 connection(s)
I20260620 11:44:53.439121 20362 server_base.cc:1166] Dumped server information to /tmp/dist-test-task1JRg7N/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781955746769051-7083-0/minicluster-data/master-0/data/info.pb
I20260620 11:44:53.440001 20430 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 11:44:53.441433 7083 external_mini_cluster.cc:1388] Started /tmp/dist-test-task1JRg7N/build/release/bin/kudu as pid 20362
I20260620 11:44:53.441566 7083 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task1JRg7N/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781955746769051-7083-0/minicluster-data/master-0/wal/instance
I20260620 11:44:53.441788 7083 external_mini_cluster.cc:1428] Setting key 194f55354e9bf41667fbdf7d77cccfaf
I20260620 11:44:53.441828 20430 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.6.234.254" port: 35005 } has no permanent_uuid. Determining permanent_uuid...
Jun 20 11:44:53 dist-test-slave-cjc6 krb5kdc[20349](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781955893, etypes {rep=17 tkt=17 ses=17}, kudu/127.6.234.254@KRBTEST.COM for kudu/127.6.234.254@KRBTEST.COM
WARNING: no policy specified for kudu/127.6.234.253@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.6.234.253@KRBTEST.COM" created.
I20260620 11:44:53.452760 20432 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 11:44:53.442659 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.234.254:35005 (local address 127.0.0.1:53156)
0620 11:44:53.442888 (+ 229us) negotiation.cc:107] Waiting for socket to connect
0620 11:44:53.442932 (+ 44us) client_negotiation.cc:174] Beginning negotiation
0620 11:44:53.443547 (+ 615us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 11:44:53.443858 (+ 311us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 11:44:53.443861 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 11:44:53.444129 (+ 268us) client_negotiation.cc:189] Negotiated authn=SASL
0620 11:44:53.444280 (+ 151us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 11:44:53.444284 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 11:44:53.445408 (+ 1124us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 11:44:53.445409 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 11:44:53.446527 (+ 1118us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 11:44:53.446530 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 11:44:53.447056 (+ 526us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 11:44:53.447057 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 11:44:53.447151 (+ 94us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 11:44:53.447656 (+ 505us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 11:44:53.447674 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0620 11:44:53.449198 (+ 1524us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 11:44:53.451041 (+ 1843us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 11:44:53.451047 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 11:44:53.451049 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0620 11:44:53.451238 (+ 189us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 11:44:53.451535 (+ 297us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 11:44:53.451536 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 11:44:53.451536 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0620 11:44:53.451651 (+ 115us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 11:44:53.452086 (+ 435us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 11:44:53.452092 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 11:44:53.452197 (+ 105us) client_negotiation.cc:769] Sending connection context
0620 11:44:53.452351 (+ 154us) client_negotiation.cc:240] Negotiation successful
0620 11:44:53.452571 (+ 220us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":149,"mutex_wait_us":86,"thread_start_us":92,"threads_started":1}
I20260620 11:44:53.452778 20433 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 11:44:53.442748 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53156 (local address 127.6.234.254:35005)
0620 11:44:53.442905 (+ 157us) server_negotiation.cc:206] Beginning negotiation
0620 11:44:53.442915 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0620 11:44:53.443037 (+ 122us) server_negotiation.cc:407] Connection header received
0620 11:44:53.443717 (+ 680us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 11:44:53.443719 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 11:44:53.443746 (+ 27us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 11:44:53.443853 (+ 107us) server_negotiation.cc:226] Negotiated authn=SASL
0620 11:44:53.444392 (+ 539us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 11:44:53.445295 (+ 903us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 11:44:53.446626 (+ 1331us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 11:44:53.446953 (+ 327us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 11:44:53.447041 (+ 88us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 11:44:53.449304 (+ 2263us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 11:44:53.449310 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 11:44:53.449311 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 11:44:53.449316 (+ 5us) server_negotiation.cc:922] calling sasl_server_start()
0620 11:44:53.450935 (+ 1619us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 11:44:53.451336 (+ 401us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 11:44:53.451339 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 11:44:53.451340 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 11:44:53.451419 (+ 79us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 11:44:53.451771 (+ 352us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 11:44:53.451772 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 11:44:53.451772 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 11:44:53.451993 (+ 221us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 11:44:53.452067 (+ 74us) server_negotiation.cc:1035] Waiting for connection context
0620 11:44:53.452382 (+ 315us) server_negotiation.cc:299] Negotiation successful
0620 11:44:53.452512 (+ 130us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":110,"thread_start_us":61,"threads_started":1}
I20260620 11:44:53.453450 20430 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.6.234.253" port: 41267 } has no permanent_uuid. Determining permanent_uuid...
I20260620 11:44:53.453819 20432 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 11:44:53.453714 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.234.253:41267 (local address 127.0.0.1:42040)
0620 11:44:53.453749 (+ 35us) negotiation.cc:107] Waiting for socket to connect
0620 11:44:53.453800 (+ 51us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.234.253:41267: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":19}
W20260620 11:44:53.453953 20381 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.6.234.253:41267: connect: Connection refused (error 111)
I20260620 11:44:53.454362 20432 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 11:44:53.454266 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.234.253:41267 (local address 127.0.0.1:42056)
0620 11:44:53.454316 (+ 50us) negotiation.cc:107] Waiting for socket to connect
0620 11:44:53.454340 (+ 24us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.234.253:41267: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":35}
W20260620 11:44:53.454470 20430 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.6.234.253:41267: Network error: Client connection negotiation failed: client connection to 127.6.234.253:41267: connect: Connection refused (error 111)
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.6.234.253 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task1JRg7N/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781955746769051-7083-0/krb5kdc/kudu_127.6.234.253.keytab.
Entry for principal kudu/127.6.234.253 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task1JRg7N/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781955746769051-7083-0/krb5kdc/kudu_127.6.234.253.keytab.
I20260620 11:44:53.469233 7083 external_mini_cluster.cc:1326] Running /tmp/dist-test-task1JRg7N/build/release/bin/kudu
/tmp/dist-test-task1JRg7N/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task1JRg7N/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781955746769051-7083-0/minicluster-data/master-1/wal
--fs_data_dirs=/tmp/dist-test-task1JRg7N/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781955746769051-7083-0/minicluster-data/master-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task1JRg7N/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781955746769051-7083-0/minicluster-data/master-1/logs
--server_dump_info_path=/tmp/dist-test-task1JRg7N/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781955746769051-7083-0/minicluster-data/master-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.6.234.253:41267
--webserver_interface=127.6.234.253
--webserver_port=0
--keytab_file=/tmp/dist-test-task1JRg7N/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781955746769051-7083-0/krb5kdc/kudu_127.6.234.253.keytab
--principal=kudu/127.6.234.253
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.6.234.212:44363
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.6.234.254:35005,127.6.234.253:41267,127.6.234.252:38097
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task1JRg7N/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781955746769051-7083-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task1JRg7N/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781955746769051-7083-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task1JRg7N/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781955746769051-7083-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260620 11:44:53.503674 20430 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.6.234.253" port: 41267 } attempt: 1
I20260620 11:44:53.504179 20432 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 11:44:53.504012 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.234.253:41267 (local address 127.0.0.1:42058)
0620 11:44:53.504086 (+ 74us) negotiation.cc:107] Waiting for socket to connect
0620 11:44:53.504138 (+ 52us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.234.253:41267: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":44}
I20260620 11:44:53.504565 20432 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 11:44:53.504464 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.234.253:41267 (local address 127.0.0.1:42066)
0620 11:44:53.504514 (+ 50us) negotiation.cc:107] Waiting for socket to connect
0620 11:44:53.504543 (+ 29us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.234.253:41267: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":34}
W20260620 11:44:53.504702 20430 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.6.234.253:41267: Network error: Client connection negotiation failed: client connection to 127.6.234.253:41267: connect: Connection refused (error 111)
W20260620 11:44:53.550684 20437 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 11:44:53.550876 20437 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 11:44:53.550897 20437 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 11:44:53.552024 20437 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 11:44:53.552066 20437 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 11:44:53.552080 20437 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 11:44:53.552127 20437 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 11:44:53.552155 20437 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 11:44:53.553828 20437 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.6.234.212:44363
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task1JRg7N/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781955746769051-7083-0/minicluster-data/master-1/data
--fs_wal_dir=/tmp/dist-test-task1JRg7N/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781955746769051-7083-0/minicluster-data/master-1/wal
--ipki_ca_key_size=768
--master_addresses=127.6.234.254:35005,127.6.234.253:41267,127.6.234.252:38097
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task1JRg7N/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781955746769051-7083-0/krb5kdc/kudu_127.6.234.253.keytab
--principal=kudu/127.6.234.253
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.6.234.253:41267
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task1JRg7N/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781955746769051-7083-0/minicluster-data/master-1/data/info.pb
--webserver_interface=127.6.234.253
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.20437
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task1JRg7N/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781955746769051-7083-0/minicluster-data/master-1/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 20 Jun 2026 10:32:37 UTC on ccycloud.quasar-bgyofv.root.comops.site
build id 4457173
I20260620 11:44:53.554080 20437 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 11:44:53.554299 20437 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 11:44:53.559130 20442 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 11:44:53.559095 20437 server_base.cc:1034] running on GCE node
W20260620 11:44:53.559079 20445 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 11:44:53.559079 20443 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 11:44:53.559585 20437 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 11:44:53.559818 20437 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 11:44:53.560962 20437 hybrid_clock.cc:648] HybridClock initialized: now 1781955893560956 us; error 33 us; skew 500 ppm
Jun 20 11:44:53 dist-test-slave-cjc6 krb5kdc[20349](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781955893, etypes {rep=17 tkt=17 ses=17}, kudu/127.6.234.253@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 11:44:53.563457 20437 init.cc:377] Logged in from keytab as kudu/127.6.234.253@KRBTEST.COM (short username kudu)
I20260620 11:44:53.563875 20437 webserver.cc:460] Webserver started at http://127.6.234.253:35095/ using document root <none> and password file <none>
I20260620 11:44:53.564109 20437 fs_manager.cc:359] Metadata directory not provided
I20260620 11:44:53.564180 20437 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 11:44:53.564339 20437 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 11:44:53.565331 20437 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task1JRg7N/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781955746769051-7083-0/minicluster-data/master-1/data/instance:
uuid: "3497f56952e84c439405a71d196b81b5"
format_stamp: "Formatted at 2026-06-20 11:44:53 on dist-test-slave-cjc6"
server_key: "12385a04babfb1efc17dad6265c0f1cf"
server_key_iv: "712a98b7b9097840ebcd9ff806d75e76"
server_key_version: "encryptionkey@0"
I20260620 11:44:53.565685 20437 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task1JRg7N/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781955746769051-7083-0/minicluster-data/master-1/wal/instance:
uuid: "3497f56952e84c439405a71d196b81b5"
format_stamp: "Formatted at 2026-06-20 11:44:53 on dist-test-slave-cjc6"
server_key: "12385a04babfb1efc17dad6265c0f1cf"
server_key_iv: "712a98b7b9097840ebcd9ff806d75e76"
server_key_version: "encryptionkey@0"
I20260620 11:44:53.566984 20437 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.001s
I20260620 11:44:53.567696 20452 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 11:44:53.567873 20437 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.001s
I20260620 11:44:53.567970 20437 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task1JRg7N/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781955746769051-7083-0/minicluster-data/master-1/data,/tmp/dist-test-task1JRg7N/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781955746769051-7083-0/minicluster-data/master-1/wal
uuid: "3497f56952e84c439405a71d196b81b5"
format_stamp: "Formatted at 2026-06-20 11:44:53 on dist-test-slave-cjc6"
server_key: "12385a04babfb1efc17dad6265c0f1cf"
server_key_iv: "712a98b7b9097840ebcd9ff806d75e76"
server_key_version: "encryptionkey@0"
I20260620 11:44:53.568051 20437 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task1JRg7N/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781955746769051-7083-0/minicluster-data/master-1/wal
metadata directory: /tmp/dist-test-task1JRg7N/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781955746769051-7083-0/minicluster-data/master-1/wal
1 data directories: /tmp/dist-test-task1JRg7N/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781955746769051-7083-0/minicluster-data/master-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 11:44:53.572873 20430 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.6.234.253" port: 41267 } attempt: 2
I20260620 11:44:53.573264 20432 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 11:44:53.573139 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.234.253:41267 (local address 127.0.0.1:42068)
0620 11:44:53.573202 (+ 63us) negotiation.cc:107] Waiting for socket to connect
0620 11:44:53.573235 (+ 33us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.234.253:41267: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":40}
I20260620 11:44:53.573630 20432 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 11:44:53.573507 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.234.253:41267 (local address 127.0.0.1:42084)
0620 11:44:53.573575 (+ 68us) negotiation.cc:107] Waiting for socket to connect
0620 11:44:53.573607 (+ 32us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.234.253:41267: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":51}
W20260620 11:44:53.573822 20430 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.6.234.253:41267: Network error: Client connection negotiation failed: client connection to 127.6.234.253:41267: connect: Connection refused (error 111)
I20260620 11:44:53.588564 20437 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 11:44:53.590853 20437 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 11:44:53.590963 20437 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 11:44:53.595008 20437 rpc_server.cc:307] RPC server started. Bound to: 127.6.234.253:41267
I20260620 11:44:53.595044 20504 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.6.234.253:41267 every 8 connection(s)
I20260620 11:44:53.595395 20437 server_base.cc:1166] Dumped server information to /tmp/dist-test-task1JRg7N/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781955746769051-7083-0/minicluster-data/master-1/data/info.pb
I20260620 11:44:53.596228 20505 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 11:44:53.597882 20505 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.6.234.254" port: 35005 } has no permanent_uuid. Determining permanent_uuid...
Jun 20 11:44:53 dist-test-slave-cjc6 krb5kdc[20349](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781955893, etypes {rep=17 tkt=17 ses=17}, kudu/127.6.234.253@KRBTEST.COM for kudu/127.6.234.254@KRBTEST.COM
I20260620 11:44:53.604342 7083 external_mini_cluster.cc:1388] Started /tmp/dist-test-task1JRg7N/build/release/bin/kudu as pid 20437
I20260620 11:44:53.604475 7083 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task1JRg7N/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781955746769051-7083-0/minicluster-data/master-1/wal/instance
I20260620 11:44:53.604672 7083 external_mini_cluster.cc:1428] Setting key 3812702e90959bc5eb5787484feadbe5
I20260620 11:44:53.607009 20433 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 11:44:53.598596 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53158 (local address 127.6.234.254:35005)
0620 11:44:53.598676 (+ 80us) server_negotiation.cc:206] Beginning negotiation
0620 11:44:53.598678 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 11:44:53.598814 (+ 136us) server_negotiation.cc:407] Connection header received
0620 11:44:53.599235 (+ 421us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 11:44:53.599236 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 11:44:53.599245 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 11:44:53.599297 (+ 52us) server_negotiation.cc:226] Negotiated authn=SASL
0620 11:44:53.600007 (+ 710us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 11:44:53.600494 (+ 487us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 11:44:53.601512 (+ 1018us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 11:44:53.601773 (+ 261us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 11:44:53.601829 (+ 56us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 11:44:53.603763 (+ 1934us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 11:44:53.603764 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 11:44:53.603764 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 11:44:53.603770 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0620 11:44:53.605515 (+ 1745us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 11:44:53.606023 (+ 508us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 11:44:53.606025 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 11:44:53.606025 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 11:44:53.606087 (+ 62us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 11:44:53.606355 (+ 268us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 11:44:53.606356 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 11:44:53.606356 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 11:44:53.606473 (+ 117us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 11:44:53.606548 (+ 75us) server_negotiation.cc:1035] Waiting for connection context
0620 11:44:53.606831 (+ 283us) server_negotiation.cc:299] Negotiation successful
0620 11:44:53.606916 (+ 85us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":50}
I20260620 11:44:53.607213 20507 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 11:44:53.598540 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.234.254:35005 (local address 127.0.0.1:53158)
0620 11:44:53.598692 (+ 152us) negotiation.cc:107] Waiting for socket to connect
0620 11:44:53.598727 (+ 35us) client_negotiation.cc:174] Beginning negotiation
0620 11:44:53.599154 (+ 427us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 11:44:53.599369 (+ 215us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 11:44:53.599373 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 11:44:53.599608 (+ 235us) client_negotiation.cc:189] Negotiated authn=SASL
0620 11:44:53.599918 (+ 310us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 11:44:53.599925 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 11:44:53.600573 (+ 648us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 11:44:53.600574 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 11:44:53.601436 (+ 862us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 11:44:53.601439 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 11:44:53.601851 (+ 412us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 11:44:53.601852 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 11:44:53.601971 (+ 119us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 11:44:53.602380 (+ 409us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 11:44:53.602388 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0620 11:44:53.603676 (+ 1288us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 11:44:53.605608 (+ 1932us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 11:44:53.605615 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 11:44:53.605617 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0620 11:44:53.605926 (+ 309us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 11:44:53.606159 (+ 233us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 11:44:53.606160 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 11:44:53.606161 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0620 11:44:53.606274 (+ 113us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 11:44:53.606552 (+ 278us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 11:44:53.606559 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 11:44:53.606696 (+ 137us) client_negotiation.cc:769] Sending connection context
0620 11:44:53.606810 (+ 114us) client_negotiation.cc:240] Negotiation successful
0620 11:44:53.606976 (+ 166us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":118,"thread_start_us":86,"threads_started":1}
I20260620 11:44:53.607795 20505 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.6.234.253" port: 41267 } has no permanent_uuid. Determining permanent_uuid...
WARNING: no policy specified for kudu/127.6.234.252@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.6.234.252@KRBTEST.COM" created.
Jun 20 11:44:53 dist-test-slave-cjc6 krb5kdc[20349](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781955893, etypes {rep=17 tkt=17 ses=17}, kudu/127.6.234.253@KRBTEST.COM for kudu/127.6.234.253@KRBTEST.COM
I20260620 11:44:53.617532 20509 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 11:44:53.608146 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42096 (local address 127.6.234.253:41267)
0620 11:44:53.608301 (+ 155us) server_negotiation.cc:206] Beginning negotiation
0620 11:44:53.608305 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 11:44:53.608320 (+ 15us) server_negotiation.cc:407] Connection header received
0620 11:44:53.608336 (+ 16us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 11:44:53.608338 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 11:44:53.608348 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 11:44:53.608435 (+ 87us) server_negotiation.cc:226] Negotiated authn=SASL
0620 11:44:53.608705 (+ 270us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 11:44:53.609374 (+ 669us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 11:44:53.610144 (+ 770us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 11:44:53.610444 (+ 300us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 11:44:53.610535 (+ 91us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 11:44:53.614652 (+ 4117us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 11:44:53.614658 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 11:44:53.614658 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 11:44:53.614664 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0620 11:44:53.616265 (+ 1601us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 11:44:53.616674 (+ 409us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 11:44:53.616676 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 11:44:53.616676 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 11:44:53.616725 (+ 49us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 11:44:53.616949 (+ 224us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 11:44:53.616950 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 11:44:53.616950 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 11:44:53.617074 (+ 124us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 11:44:53.617134 (+ 60us) server_negotiation.cc:1035] Waiting for connection context
0620 11:44:53.617302 (+ 168us) server_negotiation.cc:299] Negotiation successful
0620 11:44:53.617403 (+ 101us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":131,"thread_start_us":77,"threads_started":1}
I20260620 11:44:53.617619 20507 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 11:44:53.608056 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.234.253:41267 (local address 127.0.0.1:42096)
0620 11:44:53.608108 (+ 52us) negotiation.cc:107] Waiting for socket to connect
0620 11:44:53.608118 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0620 11:44:53.608153 (+ 35us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 11:44:53.608434 (+ 281us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 11:44:53.608434 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 11:44:53.608575 (+ 141us) client_negotiation.cc:189] Negotiated authn=SASL
0620 11:44:53.608620 (+ 45us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 11:44:53.608621 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 11:44:53.609480 (+ 859us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 11:44:53.609481 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 11:44:53.610049 (+ 568us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 11:44:53.610051 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 11:44:53.610534 (+ 483us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 11:44:53.610535 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 11:44:53.610601 (+ 66us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 11:44:53.611012 (+ 411us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 11:44:53.611017 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0620 11:44:53.614567 (+ 3550us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 11:44:53.616362 (+ 1795us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 11:44:53.616363 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 11:44:53.616363 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0620 11:44:53.616591 (+ 228us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 11:44:53.616815 (+ 224us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 11:44:53.616816 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 11:44:53.616816 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0620 11:44:53.616887 (+ 71us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 11:44:53.617146 (+ 259us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 11:44:53.617147 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 11:44:53.617196 (+ 49us) client_negotiation.cc:769] Sending connection context
0620 11:44:53.617280 (+ 84us) client_negotiation.cc:240] Negotiation successful
0620 11:44:53.617463 (+ 183us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":35,"mutex_wait_us":88}
I20260620 11:44:53.618245 20505 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.6.234.252" port: 38097 } has no permanent_uuid. Determining permanent_uuid...
I20260620 11:44:53.618580 20507 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 11:44:53.618462 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.234.252:38097 (local address 127.0.0.1:54712)
0620 11:44:53.618511 (+ 49us) negotiation.cc:107] Waiting for socket to connect
0620 11:44:53.618552 (+ 41us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.234.252:38097: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":34}
W20260620 11:44:53.618700 20455 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.6.234.252:38097: connect: Connection refused (error 111)
I20260620 11:44:53.619086 20507 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 11:44:53.618988 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.234.252:38097 (local address 127.0.0.1:54722)
0620 11:44:53.619041 (+ 53us) negotiation.cc:107] Waiting for socket to connect
0620 11:44:53.619065 (+ 24us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.234.252:38097: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":38}
W20260620 11:44:53.619189 20505 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.6.234.252:38097: Network error: Client connection negotiation failed: client connection to 127.6.234.252:38097: connect: Connection refused (error 111)
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.6.234.252 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task1JRg7N/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781955746769051-7083-0/krb5kdc/kudu_127.6.234.252.keytab.
Entry for principal kudu/127.6.234.252 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task1JRg7N/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781955746769051-7083-0/krb5kdc/kudu_127.6.234.252.keytab.
I20260620 11:44:53.628715 7083 external_mini_cluster.cc:1326] Running /tmp/dist-test-task1JRg7N/build/release/bin/kudu
/tmp/dist-test-task1JRg7N/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task1JRg7N/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781955746769051-7083-0/minicluster-data/master-2/wal
--fs_data_dirs=/tmp/dist-test-task1JRg7N/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781955746769051-7083-0/minicluster-data/master-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task1JRg7N/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781955746769051-7083-0/minicluster-data/master-2/logs
--server_dump_info_path=/tmp/dist-test-task1JRg7N/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781955746769051-7083-0/minicluster-data/master-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.6.234.252:38097
--webserver_interface=127.6.234.252
--webserver_port=0
--keytab_file=/tmp/dist-test-task1JRg7N/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781955746769051-7083-0/krb5kdc/kudu_127.6.234.252.keytab
--principal=kudu/127.6.234.252
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.6.234.212:44363
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.6.234.254:35005,127.6.234.253:41267,127.6.234.252:38097
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task1JRg7N/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781955746769051-7083-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task1JRg7N/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781955746769051-7083-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task1JRg7N/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781955746769051-7083-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260620 11:44:53.665114 20430 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.6.234.253" port: 41267 } attempt: 3
I20260620 11:44:53.668422 20505 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.6.234.252" port: 38097 } attempt: 1
I20260620 11:44:53.668838 20507 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 11:44:53.668690 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.234.252:38097 (local address 127.0.0.1:54726)
0620 11:44:53.668766 (+ 76us) negotiation.cc:107] Waiting for socket to connect
0620 11:44:53.668803 (+ 37us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.234.252:38097: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":55}
Jun 20 11:44:53 dist-test-slave-cjc6 krb5kdc[20349](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781955893, etypes {rep=17 tkt=17 ses=17}, kudu/127.6.234.254@KRBTEST.COM for kudu/127.6.234.253@KRBTEST.COM
I20260620 11:44:53.669801 20507 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 11:44:53.669666 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.234.252:38097 (local address 127.0.0.1:54732)
0620 11:44:53.669748 (+ 82us) negotiation.cc:107] Waiting for socket to connect
0620 11:44:53.669776 (+ 28us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.234.252:38097: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":67}
W20260620 11:44:53.670105 20505 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.6.234.252:38097: Network error: Client connection negotiation failed: client connection to 127.6.234.252:38097: connect: Connection refused (error 111)
I20260620 11:44:53.672541 20509 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 11:44:53.665648 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42102 (local address 127.6.234.253:41267)
0620 11:44:53.665738 (+ 90us) server_negotiation.cc:206] Beginning negotiation
0620 11:44:53.665740 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 11:44:53.665750 (+ 10us) server_negotiation.cc:407] Connection header received
0620 11:44:53.665850 (+ 100us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 11:44:53.665851 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 11:44:53.665869 (+ 18us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 11:44:53.665937 (+ 68us) server_negotiation.cc:226] Negotiated authn=SASL
0620 11:44:53.666273 (+ 336us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 11:44:53.666778 (+ 505us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 11:44:53.667466 (+ 688us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 11:44:53.667674 (+ 208us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 11:44:53.667755 (+ 81us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 11:44:53.669825 (+ 2070us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 11:44:53.669827 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 11:44:53.669827 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 11:44:53.669833 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0620 11:44:53.671356 (+ 1523us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 11:44:53.671770 (+ 414us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 11:44:53.671771 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 11:44:53.671772 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 11:44:53.671826 (+ 54us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 11:44:53.672025 (+ 199us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 11:44:53.672026 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 11:44:53.672026 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 11:44:53.672143 (+ 117us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 11:44:53.672221 (+ 78us) server_negotiation.cc:1035] Waiting for connection context
0620 11:44:53.672383 (+ 162us) server_negotiation.cc:299] Negotiation successful
0620 11:44:53.672458 (+ 75us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":61}
I20260620 11:44:53.672547 20432 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 11:44:53.665549 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.234.253:41267 (local address 127.0.0.1:42102)
0620 11:44:53.665623 (+ 74us) negotiation.cc:107] Waiting for socket to connect
0620 11:44:53.665636 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0620 11:44:53.665744 (+ 108us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 11:44:53.665943 (+ 199us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 11:44:53.665944 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 11:44:53.666112 (+ 168us) client_negotiation.cc:189] Negotiated authn=SASL
0620 11:44:53.666175 (+ 63us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 11:44:53.666177 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 11:44:53.666868 (+ 691us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 11:44:53.666869 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 11:44:53.667376 (+ 507us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 11:44:53.667378 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 11:44:53.667752 (+ 374us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 11:44:53.667753 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 11:44:53.667846 (+ 93us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 11:44:53.668324 (+ 478us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 11:44:53.668336 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0620 11:44:53.669746 (+ 1410us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 11:44:53.671452 (+ 1706us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 11:44:53.671453 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 11:44:53.671453 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0620 11:44:53.671689 (+ 236us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 11:44:53.671898 (+ 209us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 11:44:53.671899 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 11:44:53.671900 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0620 11:44:53.671957 (+ 57us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 11:44:53.672213 (+ 256us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 11:44:53.672214 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 11:44:53.672268 (+ 54us) client_negotiation.cc:769] Sending connection context
0620 11:44:53.672357 (+ 89us) client_negotiation.cc:240] Negotiation successful
0620 11:44:53.672456 (+ 99us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":47}
I20260620 11:44:53.672875 20430 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.6.234.252" port: 38097 } has no permanent_uuid. Determining permanent_uuid...
I20260620 11:44:53.673213 20432 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 11:44:53.673107 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.234.252:38097 (local address 127.0.0.1:54746)
0620 11:44:53.673159 (+ 52us) negotiation.cc:107] Waiting for socket to connect
0620 11:44:53.673184 (+ 25us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.234.252:38097: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":36}
I20260620 11:44:53.673543 20432 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 11:44:53.673457 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.234.252:38097 (local address 127.0.0.1:54750)
0620 11:44:53.673501 (+ 44us) negotiation.cc:107] Waiting for socket to connect
0620 11:44:53.673524 (+ 23us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.234.252:38097: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":32}
W20260620 11:44:53.673663 20430 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.6.234.252:38097: Network error: Client connection negotiation failed: client connection to 127.6.234.252:38097: connect: Connection refused (error 111)
I20260620 11:44:53.704823 20430 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.6.234.252" port: 38097 } attempt: 1
I20260620 11:44:53.705260 20432 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 11:44:53.705130 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.234.252:38097 (local address 127.0.0.1:54754)
0620 11:44:53.705195 (+ 65us) negotiation.cc:107] Waiting for socket to connect
0620 11:44:53.705231 (+ 36us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.234.252:38097: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":41}
I20260620 11:44:53.705799 20432 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 11:44:53.705628 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.234.252:38097 (local address 127.0.0.1:54764)
0620 11:44:53.705679 (+ 51us) negotiation.cc:107] Waiting for socket to connect
0620 11:44:53.705750 (+ 71us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.234.252:38097: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":39}
W20260620 11:44:53.705919 20430 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.6.234.252:38097: Network error: Client connection negotiation failed: client connection to 127.6.234.252:38097: connect: Connection refused (error 111)
W20260620 11:44:53.710788 20512 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 11:44:53.710989 20512 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 11:44:53.711045 20512 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 11:44:53.712178 20512 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 11:44:53.712218 20512 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 11:44:53.712234 20512 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 11:44:53.712245 20512 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 11:44:53.712302 20512 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 11:44:53.713948 20512 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.6.234.212:44363
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task1JRg7N/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781955746769051-7083-0/minicluster-data/master-2/data
--fs_wal_dir=/tmp/dist-test-task1JRg7N/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781955746769051-7083-0/minicluster-data/master-2/wal
--ipki_ca_key_size=768
--master_addresses=127.6.234.254:35005,127.6.234.253:41267,127.6.234.252:38097
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task1JRg7N/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781955746769051-7083-0/krb5kdc/kudu_127.6.234.252.keytab
--principal=kudu/127.6.234.252
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.6.234.252:38097
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task1JRg7N/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781955746769051-7083-0/minicluster-data/master-2/data/info.pb
--webserver_interface=127.6.234.252
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.20512
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task1JRg7N/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781955746769051-7083-0/minicluster-data/master-2/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 20 Jun 2026 10:32:37 UTC on ccycloud.quasar-bgyofv.root.comops.site
build id 4457173
I20260620 11:44:53.714208 20512 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 11:44:53.714447 20512 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 11:44:53.719308 20517 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 11:44:53.719290 20518 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 11:44:53.719537 20512 server_base.cc:1034] running on GCE node
W20260620 11:44:53.719290 20520 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 11:44:53.719830 20512 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 11:44:53.720142 20512 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 11:44:53.721298 20512 hybrid_clock.cc:648] HybridClock initialized: now 1781955893721289 us; error 33 us; skew 500 ppm
Jun 20 11:44:53 dist-test-slave-cjc6 krb5kdc[20349](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781955893, etypes {rep=17 tkt=17 ses=17}, kudu/127.6.234.252@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 11:44:53.724005 20512 init.cc:377] Logged in from keytab as kudu/127.6.234.252@KRBTEST.COM (short username kudu)
I20260620 11:44:53.724496 20512 webserver.cc:460] Webserver started at http://127.6.234.252:33633/ using document root <none> and password file <none>
I20260620 11:44:53.724731 20512 fs_manager.cc:359] Metadata directory not provided
I20260620 11:44:53.724802 20512 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 11:44:53.724974 20512 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 11:44:53.726135 20512 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task1JRg7N/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781955746769051-7083-0/minicluster-data/master-2/data/instance:
uuid: "03184a22d25c48fdb11b3186802d1e2e"
format_stamp: "Formatted at 2026-06-20 11:44:53 on dist-test-slave-cjc6"
server_key: "361c0c0ef18fefa3e5097191b6a34f79"
server_key_iv: "6ac48a5465aea0d842fd8b56fcbdc524"
server_key_version: "encryptionkey@0"
I20260620 11:44:53.726511 20512 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task1JRg7N/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781955746769051-7083-0/minicluster-data/master-2/wal/instance:
uuid: "03184a22d25c48fdb11b3186802d1e2e"
format_stamp: "Formatted at 2026-06-20 11:44:53 on dist-test-slave-cjc6"
server_key: "361c0c0ef18fefa3e5097191b6a34f79"
server_key_iv: "6ac48a5465aea0d842fd8b56fcbdc524"
server_key_version: "encryptionkey@0"
I20260620 11:44:53.727857 20512 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.001s sys 0.000s
I20260620 11:44:53.728670 20527 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 11:44:53.728919 20512 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260620 11:44:53.729000 20512 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task1JRg7N/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781955746769051-7083-0/minicluster-data/master-2/data,/tmp/dist-test-task1JRg7N/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781955746769051-7083-0/minicluster-data/master-2/wal
uuid: "03184a22d25c48fdb11b3186802d1e2e"
format_stamp: "Formatted at 2026-06-20 11:44:53 on dist-test-slave-cjc6"
server_key: "361c0c0ef18fefa3e5097191b6a34f79"
server_key_iv: "6ac48a5465aea0d842fd8b56fcbdc524"
server_key_version: "encryptionkey@0"
I20260620 11:44:53.729110 20512 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task1JRg7N/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781955746769051-7083-0/minicluster-data/master-2/wal
metadata directory: /tmp/dist-test-task1JRg7N/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781955746769051-7083-0/minicluster-data/master-2/wal
1 data directories: /tmp/dist-test-task1JRg7N/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781955746769051-7083-0/minicluster-data/master-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 11:44:53.738263 20505 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.6.234.252" port: 38097 } attempt: 2
I20260620 11:44:53.738698 20507 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 11:44:53.738543 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.234.252:38097 (local address 127.0.0.1:54768)
0620 11:44:53.738625 (+ 82us) negotiation.cc:107] Waiting for socket to connect
0620 11:44:53.738667 (+ 42us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.234.252:38097: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":56}
I20260620 11:44:53.739147 20507 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 11:44:53.739030 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.234.252:38097 (local address 127.0.0.1:54784)
0620 11:44:53.739089 (+ 59us) negotiation.cc:107] Waiting for socket to connect
0620 11:44:53.739125 (+ 36us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.234.252:38097: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":40}
W20260620 11:44:53.739274 20505 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.6.234.252:38097: Network error: Client connection negotiation failed: client connection to 127.6.234.252:38097: connect: Connection refused (error 111)
I20260620 11:44:53.744899 20512 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 11:44:53.747226 20512 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 11:44:53.747383 20512 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 11:44:53.751458 20512 rpc_server.cc:307] RPC server started. Bound to: 127.6.234.252:38097
I20260620 11:44:53.751487 20579 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.6.234.252:38097 every 8 connection(s)
I20260620 11:44:53.751827 20512 server_base.cc:1166] Dumped server information to /tmp/dist-test-task1JRg7N/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781955746769051-7083-0/minicluster-data/master-2/data/info.pb
I20260620 11:44:53.752638 20580 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 11:44:53.753319 7083 external_mini_cluster.cc:1388] Started /tmp/dist-test-task1JRg7N/build/release/bin/kudu as pid 20512
I20260620 11:44:53.753453 7083 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task1JRg7N/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781955746769051-7083-0/minicluster-data/master-2/wal/instance
I20260620 11:44:53.753656 7083 external_mini_cluster.cc:1428] Setting key 1c362624dba5c589cf235bbb9c896553
I20260620 11:44:53.754727 20580 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.6.234.254" port: 35005 } has no permanent_uuid. Determining permanent_uuid...
Jun 20 11:44:53 dist-test-slave-cjc6 krb5kdc[20349](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781955893, etypes {rep=17 tkt=17 ses=17}, kudu/127.6.234.252@KRBTEST.COM for kudu/127.6.234.254@KRBTEST.COM
Jun 20 11:44:53 dist-test-slave-cjc6 krb5kdc[20349](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781955893, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.6.234.252@KRBTEST.COM
I20260620 11:44:53.763370 20433 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 11:44:53.755610 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53164 (local address 127.6.234.254:35005)
0620 11:44:53.755659 (+ 49us) server_negotiation.cc:206] Beginning negotiation
0620 11:44:53.755661 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 11:44:53.755863 (+ 202us) server_negotiation.cc:407] Connection header received
0620 11:44:53.755944 (+ 81us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 11:44:53.755945 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 11:44:53.755956 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 11:44:53.756013 (+ 57us) server_negotiation.cc:226] Negotiated authn=SASL
0620 11:44:53.756499 (+ 486us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 11:44:53.756962 (+ 463us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 11:44:53.757944 (+ 982us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 11:44:53.758159 (+ 215us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 11:44:53.758252 (+ 93us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 11:44:53.760242 (+ 1990us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 11:44:53.760243 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 11:44:53.760244 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 11:44:53.760251 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0620 11:44:53.761904 (+ 1653us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 11:44:53.762390 (+ 486us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 11:44:53.762391 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 11:44:53.762392 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 11:44:53.762467 (+ 75us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 11:44:53.762745 (+ 278us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 11:44:53.762746 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 11:44:53.762747 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 11:44:53.762864 (+ 117us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 11:44:53.762944 (+ 80us) server_negotiation.cc:1035] Waiting for connection context
0620 11:44:53.763190 (+ 246us) server_negotiation.cc:299] Negotiation successful
0620 11:44:53.763273 (+ 83us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":26}
I20260620 11:44:53.763592 20583 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 11:44:53.754213 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54798 (local address 127.6.234.252:38097)
0620 11:44:53.754406 (+ 193us) server_negotiation.cc:206] Beginning negotiation
0620 11:44:53.754415 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0620 11:44:53.754447 (+ 32us) server_negotiation.cc:407] Connection header received
0620 11:44:53.754978 (+ 531us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 11:44:53.754986 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 11:44:53.755049 (+ 63us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 11:44:53.755197 (+ 148us) server_negotiation.cc:226] Negotiated authn=SASL
0620 11:44:53.755963 (+ 766us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 11:44:53.756805 (+ 842us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 11:44:53.757557 (+ 752us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 11:44:53.757983 (+ 426us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 11:44:53.758109 (+ 126us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 11:44:53.760670 (+ 2561us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 11:44:53.760677 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 11:44:53.760677 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 11:44:53.760684 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0620 11:44:53.762259 (+ 1575us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 11:44:53.762621 (+ 362us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 11:44:53.762624 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 11:44:53.762625 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 11:44:53.762680 (+ 55us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 11:44:53.762878 (+ 198us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 11:44:53.762879 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 11:44:53.762879 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 11:44:53.763064 (+ 185us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 11:44:53.763209 (+ 145us) server_negotiation.cc:1035] Waiting for connection context
0620 11:44:53.763295 (+ 86us) server_negotiation.cc:299] Negotiation successful
0620 11:44:53.763388 (+ 93us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":48,"server-negotiator.queue_time_us":142,"thread_start_us":86,"threads_started":1}
I20260620 11:44:53.763597 20584 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 11:44:53.755567 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.234.254:35005 (local address 127.0.0.1:53164)
0620 11:44:53.755771 (+ 204us) negotiation.cc:107] Waiting for socket to connect
0620 11:44:53.755798 (+ 27us) client_negotiation.cc:174] Beginning negotiation
0620 11:44:53.755880 (+ 82us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 11:44:53.756031 (+ 151us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 11:44:53.756034 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 11:44:53.756295 (+ 261us) client_negotiation.cc:189] Negotiated authn=SASL
0620 11:44:53.756404 (+ 109us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 11:44:53.756409 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 11:44:53.757053 (+ 644us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 11:44:53.757054 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 11:44:53.757847 (+ 793us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 11:44:53.757849 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 11:44:53.758260 (+ 411us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 11:44:53.758261 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 11:44:53.758339 (+ 78us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 11:44:53.758792 (+ 453us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 11:44:53.758799 (+ 7us) client_negotiation.cc:656] calling sasl_client_start()
0620 11:44:53.760151 (+ 1352us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 11:44:53.762010 (+ 1859us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 11:44:53.762018 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 11:44:53.762020 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0620 11:44:53.762298 (+ 278us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 11:44:53.762555 (+ 257us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 11:44:53.762556 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 11:44:53.762557 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0620 11:44:53.762671 (+ 114us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 11:44:53.762936 (+ 265us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 11:44:53.762943 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 11:44:53.763069 (+ 126us) client_negotiation.cc:769] Sending connection context
0620 11:44:53.763170 (+ 101us) client_negotiation.cc:240] Negotiation successful
0620 11:44:53.763335 (+ 165us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":173,"mutex_wait_us":1,"thread_start_us":68,"threads_started":1}
I20260620 11:44:53.764369 20580 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.6.234.253" port: 41267 } has no permanent_uuid. Determining permanent_uuid...
Jun 20 11:44:53 dist-test-slave-cjc6 krb5kdc[20349](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781955893, etypes {rep=17 tkt=17 ses=17}, kudu/127.6.234.252@KRBTEST.COM for kudu/127.6.234.253@KRBTEST.COM
Jun 20 11:44:53 dist-test-slave-cjc6 krb5kdc[20349](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781955893, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.6.234.254@KRBTEST.COM
I20260620 11:44:53.772183 20584 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 11:44:53.764620 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.234.253:41267 (local address 127.0.0.1:42108)
0620 11:44:53.764659 (+ 39us) negotiation.cc:107] Waiting for socket to connect
0620 11:44:53.764667 (+ 8us) client_negotiation.cc:174] Beginning negotiation
0620 11:44:53.764695 (+ 28us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 11:44:53.764909 (+ 214us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 11:44:53.764910 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 11:44:53.765059 (+ 149us) client_negotiation.cc:189] Negotiated authn=SASL
0620 11:44:53.765101 (+ 42us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 11:44:53.765103 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 11:44:53.766181 (+ 1078us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 11:44:53.766182 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 11:44:53.766949 (+ 767us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 11:44:53.766952 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 11:44:53.767449 (+ 497us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 11:44:53.767450 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 11:44:53.767516 (+ 66us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 11:44:53.767888 (+ 372us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 11:44:53.767893 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0620 11:44:53.769239 (+ 1346us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 11:44:53.771099 (+ 1860us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 11:44:53.771100 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 11:44:53.771101 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0620 11:44:53.771352 (+ 251us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 11:44:53.771550 (+ 198us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 11:44:53.771550 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 11:44:53.771551 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0620 11:44:53.771606 (+ 55us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 11:44:53.771927 (+ 321us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 11:44:53.771928 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 11:44:53.771982 (+ 54us) client_negotiation.cc:769] Sending connection context
0620 11:44:53.772044 (+ 62us) client_negotiation.cc:240] Negotiation successful
0620 11:44:53.772120 (+ 76us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":27}
I20260620 11:44:53.772274 20509 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 11:44:53.764726 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42108 (local address 127.6.234.253:41267)
0620 11:44:53.764795 (+ 69us) server_negotiation.cc:206] Beginning negotiation
0620 11:44:53.764797 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 11:44:53.764805 (+ 8us) server_negotiation.cc:407] Connection header received
0620 11:44:53.764828 (+ 23us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 11:44:53.764829 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 11:44:53.764837 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 11:44:53.764894 (+ 57us) server_negotiation.cc:226] Negotiated authn=SASL
0620 11:44:53.765225 (+ 331us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 11:44:53.766079 (+ 854us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 11:44:53.767045 (+ 966us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 11:44:53.767371 (+ 326us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 11:44:53.767447 (+ 76us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 11:44:53.769331 (+ 1884us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 11:44:53.769332 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 11:44:53.769332 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 11:44:53.769340 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0620 11:44:53.771017 (+ 1677us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 11:44:53.771432 (+ 415us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 11:44:53.771433 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 11:44:53.771434 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 11:44:53.771492 (+ 58us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 11:44:53.771672 (+ 180us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 11:44:53.771673 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 11:44:53.771674 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 11:44:53.771810 (+ 136us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 11:44:53.771876 (+ 66us) server_negotiation.cc:1035] Waiting for connection context
0620 11:44:53.772092 (+ 216us) server_negotiation.cc:299] Negotiation successful
0620 11:44:53.772165 (+ 73us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":45}
I20260620 11:44:53.772763 20580 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.6.234.252" port: 38097 } has no permanent_uuid. Determining permanent_uuid...
I20260620 11:44:53.773754 20433 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 11:44:53.764483 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53170 (local address 127.6.234.254:35005)
0620 11:44:53.764523 (+ 40us) server_negotiation.cc:206] Beginning negotiation
0620 11:44:53.764525 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 11:44:53.764569 (+ 44us) server_negotiation.cc:407] Connection header received
0620 11:44:53.764644 (+ 75us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 11:44:53.764645 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 11:44:53.764655 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 11:44:53.764786 (+ 131us) server_negotiation.cc:226] Negotiated authn=SASL
0620 11:44:53.765355 (+ 569us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 11:44:53.766124 (+ 769us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 11:44:53.767121 (+ 997us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 11:44:53.767446 (+ 325us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 11:44:53.767513 (+ 67us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 11:44:53.770380 (+ 2867us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 11:44:53.770381 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 11:44:53.770382 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 11:44:53.770389 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0620 11:44:53.772215 (+ 1826us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 11:44:53.772679 (+ 464us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 11:44:53.772680 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 11:44:53.772680 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 11:44:53.772739 (+ 59us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 11:44:53.773066 (+ 327us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 11:44:53.773067 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 11:44:53.773067 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 11:44:53.773218 (+ 151us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 11:44:53.773382 (+ 164us) server_negotiation.cc:1035] Waiting for connection context
0620 11:44:53.773573 (+ 191us) server_negotiation.cc:299] Negotiation successful
0620 11:44:53.773633 (+ 60us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":19}
Jun 20 11:44:53 dist-test-slave-cjc6 krb5kdc[20349](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781955893, etypes {rep=17 tkt=17 ses=17}, kudu/127.6.234.252@KRBTEST.COM for kudu/127.6.234.252@KRBTEST.COM
Jun 20 11:44:53 dist-test-slave-cjc6 krb5kdc[20349](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781955893, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.6.234.253@KRBTEST.COM
I20260620 11:44:53.780427 20583 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 11:44:53.773103 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54812 (local address 127.6.234.252:38097)
0620 11:44:53.773157 (+ 54us) server_negotiation.cc:206] Beginning negotiation
0620 11:44:53.773159 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 11:44:53.773169 (+ 10us) server_negotiation.cc:407] Connection header received
0620 11:44:53.773193 (+ 24us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 11:44:53.773194 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 11:44:53.773202 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 11:44:53.773267 (+ 65us) server_negotiation.cc:226] Negotiated authn=SASL
0620 11:44:53.773478 (+ 211us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 11:44:53.774178 (+ 700us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 11:44:53.774994 (+ 816us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 11:44:53.775247 (+ 253us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 11:44:53.775296 (+ 49us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 11:44:53.777409 (+ 2113us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 11:44:53.777411 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 11:44:53.777411 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 11:44:53.777418 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0620 11:44:53.779184 (+ 1766us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 11:44:53.779559 (+ 375us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 11:44:53.779560 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 11:44:53.779561 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 11:44:53.779616 (+ 55us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 11:44:53.779862 (+ 246us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 11:44:53.779863 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 11:44:53.779864 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 11:44:53.779979 (+ 115us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 11:44:53.780059 (+ 80us) server_negotiation.cc:1035] Waiting for connection context
0620 11:44:53.780219 (+ 160us) server_negotiation.cc:299] Negotiation successful
0620 11:44:53.780313 (+ 94us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":2,"server-negotiator.queue_time_us":36}
I20260620 11:44:53.780517 20584 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 11:44:53.773018 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.234.252:38097 (local address 127.0.0.1:54812)
0620 11:44:53.773069 (+ 51us) negotiation.cc:107] Waiting for socket to connect
0620 11:44:53.773083 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0620 11:44:53.773129 (+ 46us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 11:44:53.773270 (+ 141us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 11:44:53.773270 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 11:44:53.773381 (+ 111us) client_negotiation.cc:189] Negotiated authn=SASL
0620 11:44:53.773408 (+ 27us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 11:44:53.773409 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 11:44:53.774421 (+ 1012us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 11:44:53.774422 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 11:44:53.774916 (+ 494us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 11:44:53.774917 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 11:44:53.775329 (+ 412us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 11:44:53.775330 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 11:44:53.775388 (+ 58us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 11:44:53.775876 (+ 488us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 11:44:53.775880 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0620 11:44:53.777314 (+ 1434us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 11:44:53.779287 (+ 1973us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 11:44:53.779288 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 11:44:53.779288 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0620 11:44:53.779463 (+ 175us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 11:44:53.779704 (+ 241us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 11:44:53.779705 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 11:44:53.779705 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0620 11:44:53.779769 (+ 64us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 11:44:53.780051 (+ 282us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 11:44:53.780051 (+ 0us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 11:44:53.780090 (+ 39us) client_negotiation.cc:769] Sending connection context
0620 11:44:53.780192 (+ 102us) client_negotiation.cc:240] Negotiation successful
0620 11:44:53.780387 (+ 195us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":34,"mutex_wait_us":94}
I20260620 11:44:53.781091 20430 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.6.234.252" port: 38097 } attempt: 2
I20260620 11:44:53.782094 20509 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 11:44:53.774391 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42110 (local address 127.6.234.253:41267)
0620 11:44:53.774446 (+ 55us) server_negotiation.cc:206] Beginning negotiation
0620 11:44:53.774448 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 11:44:53.774489 (+ 41us) server_negotiation.cc:407] Connection header received
0620 11:44:53.774555 (+ 66us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 11:44:53.774556 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 11:44:53.774565 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 11:44:53.774617 (+ 52us) server_negotiation.cc:226] Negotiated authn=SASL
0620 11:44:53.775062 (+ 445us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 11:44:53.775679 (+ 617us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 11:44:53.776441 (+ 762us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 11:44:53.776628 (+ 187us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 11:44:53.776686 (+ 58us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 11:44:53.778846 (+ 2160us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 11:44:53.778847 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 11:44:53.778847 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 11:44:53.778856 (+ 9us) server_negotiation.cc:922] calling sasl_server_start()
0620 11:44:53.780342 (+ 1486us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 11:44:53.780775 (+ 433us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 11:44:53.780776 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 11:44:53.780776 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 11:44:53.780843 (+ 67us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 11:44:53.781098 (+ 255us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 11:44:53.781099 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 11:44:53.781099 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 11:44:53.781252 (+ 153us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 11:44:53.781446 (+ 194us) server_negotiation.cc:1035] Waiting for connection context
0620 11:44:53.781894 (+ 448us) server_negotiation.cc:299] Negotiation successful
0620 11:44:53.781975 (+ 81us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":36}
I20260620 11:44:53.782569 20580 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 03184a22d25c48fdb11b3186802d1e2e: Bootstrap starting.
I20260620 11:44:53.783560 7083 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260620 11:44:53.783730 20580 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 03184a22d25c48fdb11b3186802d1e2e: Neither blocks nor log segments found. Creating new log.
I20260620 11:44:53.784065 20580 log.cc:824] T 00000000000000000000000000000000 P 03184a22d25c48fdb11b3186802d1e2e: Log is configured to *not* fsync() on all Append() calls
I20260620 11:44:53.785156 20580 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 03184a22d25c48fdb11b3186802d1e2e: No bootstrap required, opened a new log
Jun 20 11:44:53 dist-test-slave-cjc6 krb5kdc[20349](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781955893, etypes {rep=17 tkt=17 ses=17}, kudu/127.6.234.254@KRBTEST.COM for kudu/127.6.234.252@KRBTEST.COM
I20260620 11:44:53.787595 20580 raft_consensus.cc:348] T 00000000000000000000000000000000 P 03184a22d25c48fdb11b3186802d1e2e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "619c5ff12b6949eea40c74e81640db02" member_type: VOTER last_known_addr { host: "127.6.234.254" port: 35005 } } peers { permanent_uuid: "3497f56952e84c439405a71d196b81b5" member_type: VOTER last_known_addr { host: "127.6.234.253" port: 41267 } } peers { permanent_uuid: "03184a22d25c48fdb11b3186802d1e2e" member_type: VOTER last_known_addr { host: "127.6.234.252" port: 38097 } }
I20260620 11:44:53.787757 20580 raft_consensus.cc:374] T 00000000000000000000000000000000 P 03184a22d25c48fdb11b3186802d1e2e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 11:44:53.787819 20580 raft_consensus.cc:729] T 00000000000000000000000000000000 P 03184a22d25c48fdb11b3186802d1e2e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 03184a22d25c48fdb11b3186802d1e2e, State: Initialized, Role: FOLLOWER
I20260620 11:44:53.788007 20580 consensus_queue.cc:260] T 00000000000000000000000000000000 P 03184a22d25c48fdb11b3186802d1e2e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "619c5ff12b6949eea40c74e81640db02" member_type: VOTER last_known_addr { host: "127.6.234.254" port: 35005 } } peers { permanent_uuid: "3497f56952e84c439405a71d196b81b5" member_type: VOTER last_known_addr { host: "127.6.234.253" port: 41267 } } peers { permanent_uuid: "03184a22d25c48fdb11b3186802d1e2e" member_type: VOTER last_known_addr { host: "127.6.234.252" port: 38097 } }
I20260620 11:44:53.788403 20580 sys_catalog.cc:564] T 00000000000000000000000000000000 P 03184a22d25c48fdb11b3186802d1e2e [sys.catalog]: configured and running, proceeding with master startup.
I20260620 11:44:53.789849 20432 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 11:44:53.781361 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.234.252:38097 (local address 127.0.0.1:54826)
0620 11:44:53.781409 (+ 48us) negotiation.cc:107] Waiting for socket to connect
0620 11:44:53.781420 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0620 11:44:53.781474 (+ 54us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 11:44:53.781614 (+ 140us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 11:44:53.781615 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 11:44:53.781853 (+ 238us) client_negotiation.cc:189] Negotiated authn=SASL
0620 11:44:53.781901 (+ 48us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 11:44:53.781903 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 11:44:53.782879 (+ 976us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 11:44:53.782880 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 11:44:53.783795 (+ 915us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 11:44:53.783798 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 11:44:53.784320 (+ 522us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 11:44:53.784321 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 11:44:53.784383 (+ 62us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 11:44:53.784831 (+ 448us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 11:44:53.784837 (+ 6us) client_negotiation.cc:656] calling sasl_client_start()
0620 11:44:53.786553 (+ 1716us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 11:44:53.788562 (+ 2009us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 11:44:53.788563 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 11:44:53.788563 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0620 11:44:53.788804 (+ 241us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 11:44:53.789005 (+ 201us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 11:44:53.789006 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 11:44:53.789007 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0620 11:44:53.789065 (+ 58us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 11:44:53.789426 (+ 361us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 11:44:53.789427 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 11:44:53.789475 (+ 48us) client_negotiation.cc:769] Sending connection context
0620 11:44:53.789547 (+ 72us) client_negotiation.cc:240] Negotiation successful
0620 11:44:53.789656 (+ 109us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":33}
I20260620 11:44:53.790190 20583 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 11:44:53.781395 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54826 (local address 127.6.234.252:38097)
0620 11:44:53.781429 (+ 34us) server_negotiation.cc:206] Beginning negotiation
0620 11:44:53.781431 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 11:44:53.781453 (+ 22us) server_negotiation.cc:407] Connection header received
0620 11:44:53.781545 (+ 92us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 11:44:53.781546 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 11:44:53.781556 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 11:44:53.781599 (+ 43us) server_negotiation.cc:226] Negotiated authn=SASL
0620 11:44:53.781959 (+ 360us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 11:44:53.782794 (+ 835us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 11:44:53.783910 (+ 1116us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 11:44:53.784232 (+ 322us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 11:44:53.784312 (+ 80us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 11:44:53.786646 (+ 2334us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 11:44:53.786648 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 11:44:53.786648 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 11:44:53.786655 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0620 11:44:53.788462 (+ 1807us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 11:44:53.788881 (+ 419us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 11:44:53.788882 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 11:44:53.788882 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 11:44:53.788938 (+ 56us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 11:44:53.789136 (+ 198us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 11:44:53.789137 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 11:44:53.789137 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 11:44:53.789319 (+ 182us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 11:44:53.789395 (+ 76us) server_negotiation.cc:1035] Waiting for connection context
0620 11:44:53.789988 (+ 593us) server_negotiation.cc:299] Negotiation successful
0620 11:44:53.790079 (+ 91us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":18}
I20260620 11:44:53.791694 20430 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 619c5ff12b6949eea40c74e81640db02: Bootstrap starting.
I20260620 11:44:53.792423 20430 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 619c5ff12b6949eea40c74e81640db02: Neither blocks nor log segments found. Creating new log.
I20260620 11:44:53.792784 20430 log.cc:824] T 00000000000000000000000000000000 P 619c5ff12b6949eea40c74e81640db02: Log is configured to *not* fsync() on all Append() calls
I20260620 11:44:53.790577 20512 master_runner.cc:186] Error getting master registration for 127.6.234.254:35005: OK, instance_id { permanent_uuid: "619c5ff12b6949eea40c74e81640db02" instance_seqno: 1781955893437480 } error { code: CATALOG_MANAGER_NOT_INITIALIZED status { code: SERVICE_UNAVAILABLE message: "Catalog manager is not initialized. State: Starting" } }
I20260620 11:44:53.793609 20512 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20260620 11:44:53.793787 20430 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 619c5ff12b6949eea40c74e81640db02: No bootstrap required, opened a new log
W20260620 11:44:53.794597 20605 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 03184a22d25c48fdb11b3186802d1e2e: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260620 11:44:53.794642 20605 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260620 11:44:53.794792 20509 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 11:44:53.785040 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42114 (local address 127.6.234.253:41267)
0620 11:44:53.785080 (+ 40us) server_negotiation.cc:206] Beginning negotiation
0620 11:44:53.785082 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 11:44:53.785157 (+ 75us) server_negotiation.cc:407] Connection header received
0620 11:44:53.785227 (+ 70us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 11:44:53.785228 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 11:44:53.785238 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 11:44:53.785299 (+ 61us) server_negotiation.cc:226] Negotiated authn=SASL
0620 11:44:53.786174 (+ 875us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 11:44:53.787258 (+ 1084us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 11:44:53.788771 (+ 1513us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 11:44:53.789095 (+ 324us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 11:44:53.789140 (+ 45us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 11:44:53.791079 (+ 1939us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 11:44:53.791080 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 11:44:53.791081 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 11:44:53.791089 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0620 11:44:53.792625 (+ 1536us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 11:44:53.793041 (+ 416us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 11:44:53.793042 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 11:44:53.793042 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 11:44:53.793110 (+ 68us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 11:44:53.793370 (+ 260us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 11:44:53.793371 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 11:44:53.793372 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 11:44:53.793515 (+ 143us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 11:44:53.794188 (+ 673us) server_negotiation.cc:1035] Waiting for connection context
0620 11:44:53.794542 (+ 354us) server_negotiation.cc:299] Negotiation successful
0620 11:44:53.794688 (+ 146us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":21}
I20260620 11:44:53.794791 20594 sys_catalog.cc:455] T 00000000000000000000000000000000 P 03184a22d25c48fdb11b3186802d1e2e [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "619c5ff12b6949eea40c74e81640db02" member_type: VOTER last_known_addr { host: "127.6.234.254" port: 35005 } } peers { permanent_uuid: "3497f56952e84c439405a71d196b81b5" member_type: VOTER last_known_addr { host: "127.6.234.253" port: 41267 } } peers { permanent_uuid: "03184a22d25c48fdb11b3186802d1e2e" member_type: VOTER last_known_addr { host: "127.6.234.252" port: 38097 } } }
I20260620 11:44:53.794870 20594 sys_catalog.cc:458] T 00000000000000000000000000000000 P 03184a22d25c48fdb11b3186802d1e2e [sys.catalog]: This master's current role is: FOLLOWER
I20260620 11:44:53.796734 20430 raft_consensus.cc:348] T 00000000000000000000000000000000 P 619c5ff12b6949eea40c74e81640db02 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "619c5ff12b6949eea40c74e81640db02" member_type: VOTER last_known_addr { host: "127.6.234.254" port: 35005 } } peers { permanent_uuid: "3497f56952e84c439405a71d196b81b5" member_type: VOTER last_known_addr { host: "127.6.234.253" port: 41267 } } peers { permanent_uuid: "03184a22d25c48fdb11b3186802d1e2e" member_type: VOTER last_known_addr { host: "127.6.234.252" port: 38097 } }
I20260620 11:44:53.796972 20433 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 11:44:53.785377 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53178 (local address 127.6.234.254:35005)
0620 11:44:53.785412 (+ 35us) server_negotiation.cc:206] Beginning negotiation
0620 11:44:53.785414 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 11:44:53.786023 (+ 609us) server_negotiation.cc:407] Connection header received
0620 11:44:53.786112 (+ 89us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 11:44:53.786113 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 11:44:53.786123 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 11:44:53.786153 (+ 30us) server_negotiation.cc:226] Negotiated authn=SASL
0620 11:44:53.786854 (+ 701us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 11:44:53.787509 (+ 655us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 11:44:53.788668 (+ 1159us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 11:44:53.789002 (+ 334us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 11:44:53.789052 (+ 50us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 11:44:53.793581 (+ 4529us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 11:44:53.793583 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 11:44:53.793583 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 11:44:53.793590 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0620 11:44:53.795410 (+ 1820us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 11:44:53.795986 (+ 576us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 11:44:53.795987 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 11:44:53.795988 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 11:44:53.796054 (+ 66us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 11:44:53.796308 (+ 254us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 11:44:53.796309 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 11:44:53.796310 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 11:44:53.796456 (+ 146us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 11:44:53.796565 (+ 109us) server_negotiation.cc:1035] Waiting for connection context
0620 11:44:53.796784 (+ 219us) server_negotiation.cc:299] Negotiation successful
0620 11:44:53.796868 (+ 84us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":18}
I20260620 11:44:53.796949 20430 raft_consensus.cc:374] T 00000000000000000000000000000000 P 619c5ff12b6949eea40c74e81640db02 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 11:44:53.797345 20430 raft_consensus.cc:729] T 00000000000000000000000000000000 P 619c5ff12b6949eea40c74e81640db02 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 619c5ff12b6949eea40c74e81640db02, State: Initialized, Role: FOLLOWER
I20260620 11:44:53.797569 20430 consensus_queue.cc:260] T 00000000000000000000000000000000 P 619c5ff12b6949eea40c74e81640db02 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "619c5ff12b6949eea40c74e81640db02" member_type: VOTER last_known_addr { host: "127.6.234.254" port: 35005 } } peers { permanent_uuid: "3497f56952e84c439405a71d196b81b5" member_type: VOTER last_known_addr { host: "127.6.234.253" port: 41267 } } peers { permanent_uuid: "03184a22d25c48fdb11b3186802d1e2e" member_type: VOTER last_known_addr { host: "127.6.234.252" port: 38097 } }
I20260620 11:44:53.797989 20608 sys_catalog.cc:455] T 00000000000000000000000000000000 P 619c5ff12b6949eea40c74e81640db02 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "619c5ff12b6949eea40c74e81640db02" member_type: VOTER last_known_addr { host: "127.6.234.254" port: 35005 } } peers { permanent_uuid: "3497f56952e84c439405a71d196b81b5" member_type: VOTER last_known_addr { host: "127.6.234.253" port: 41267 } } peers { permanent_uuid: "03184a22d25c48fdb11b3186802d1e2e" member_type: VOTER last_known_addr { host: "127.6.234.252" port: 38097 } } }
I20260620 11:44:53.798059 20430 sys_catalog.cc:564] T 00000000000000000000000000000000 P 619c5ff12b6949eea40c74e81640db02 [sys.catalog]: configured and running, proceeding with master startup.
I20260620 11:44:53.798138 20608 sys_catalog.cc:458] T 00000000000000000000000000000000 P 619c5ff12b6949eea40c74e81640db02 [sys.catalog]: This master's current role is: FOLLOWER
W20260620 11:44:53.800113 20619 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 619c5ff12b6949eea40c74e81640db02: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260620 11:44:53.800203 20619 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260620 11:44:53.803284 20592 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 11:44:53.785668 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54832 (local address 127.6.234.252:38097)
0620 11:44:53.785923 (+ 255us) server_negotiation.cc:206] Beginning negotiation
0620 11:44:53.785925 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 11:44:53.793965 (+ 8040us) server_negotiation.cc:407] Connection header received
0620 11:44:53.793991 (+ 26us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 11:44:53.793992 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 11:44:53.794004 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 11:44:53.794070 (+ 66us) server_negotiation.cc:226] Negotiated authn=SASL
0620 11:44:53.796362 (+ 2292us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 11:44:53.797100 (+ 738us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 11:44:53.798101 (+ 1001us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 11:44:53.798415 (+ 314us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 11:44:53.798459 (+ 44us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 11:44:53.800004 (+ 1545us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 11:44:53.800005 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 11:44:53.800006 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 11:44:53.800013 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0620 11:44:53.801797 (+ 1784us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 11:44:53.802167 (+ 370us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 11:44:53.802168 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 11:44:53.802168 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 11:44:53.802247 (+ 79us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 11:44:53.802441 (+ 194us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 11:44:53.802441 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 11:44:53.802442 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 11:44:53.802561 (+ 119us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 11:44:53.802656 (+ 95us) server_negotiation.cc:1035] Waiting for connection context
0620 11:44:53.803069 (+ 413us) server_negotiation.cc:299] Negotiation successful
0620 11:44:53.803146 (+ 77us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":235,"thread_start_us":91,"threads_started":1}
I20260620 11:44:53.801465 20362 master_runner.cc:186] Error getting master registration for 127.6.234.253:41267: OK, instance_id { permanent_uuid: "3497f56952e84c439405a71d196b81b5" instance_seqno: 1781955893593749 } error { code: CATALOG_MANAGER_NOT_INITIALIZED status { code: SERVICE_UNAVAILABLE message: "Catalog manager is not initialized. State: Starting" } }
I20260620 11:44:53.804615 20362 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20260620 11:44:53.830438 20505 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.6.234.252" port: 38097 } attempt: 3
Jun 20 11:44:53 dist-test-slave-cjc6 krb5kdc[20349](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781955893, etypes {rep=17 tkt=17 ses=17}, kudu/127.6.234.253@KRBTEST.COM for kudu/127.6.234.252@KRBTEST.COM
I20260620 11:44:53.839157 20507 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 11:44:53.830750 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.234.252:38097 (local address 127.0.0.1:54838)
0620 11:44:53.830803 (+ 53us) negotiation.cc:107] Waiting for socket to connect
0620 11:44:53.830824 (+ 21us) client_negotiation.cc:174] Beginning negotiation
0620 11:44:53.830864 (+ 40us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 11:44:53.831019 (+ 155us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 11:44:53.831020 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 11:44:53.831233 (+ 213us) client_negotiation.cc:189] Negotiated authn=SASL
0620 11:44:53.831293 (+ 60us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 11:44:53.831295 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 11:44:53.832283 (+ 988us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 11:44:53.832284 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 11:44:53.833204 (+ 920us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 11:44:53.833207 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 11:44:53.833773 (+ 566us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 11:44:53.833775 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 11:44:53.833843 (+ 68us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 11:44:53.834378 (+ 535us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 11:44:53.834384 (+ 6us) client_negotiation.cc:656] calling sasl_client_start()
0620 11:44:53.835994 (+ 1610us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 11:44:53.837921 (+ 1927us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 11:44:53.837922 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 11:44:53.837922 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0620 11:44:53.838165 (+ 243us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 11:44:53.838404 (+ 239us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 11:44:53.838405 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 11:44:53.838405 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0620 11:44:53.838477 (+ 72us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 11:44:53.838785 (+ 308us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 11:44:53.838786 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 11:44:53.838839 (+ 53us) client_negotiation.cc:769] Sending connection context
0620 11:44:53.838920 (+ 81us) client_negotiation.cc:240] Negotiation successful
0620 11:44:53.839024 (+ 104us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":33}
I20260620 11:44:53.839154 20592 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 11:44:53.830812 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54838 (local address 127.6.234.252:38097)
0620 11:44:53.830869 (+ 57us) server_negotiation.cc:206] Beginning negotiation
0620 11:44:53.830871 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 11:44:53.830905 (+ 34us) server_negotiation.cc:407] Connection header received
0620 11:44:53.830927 (+ 22us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 11:44:53.830928 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 11:44:53.830938 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 11:44:53.830991 (+ 53us) server_negotiation.cc:226] Negotiated authn=SASL
0620 11:44:53.831382 (+ 391us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 11:44:53.832197 (+ 815us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 11:44:53.833304 (+ 1107us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 11:44:53.833659 (+ 355us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 11:44:53.833744 (+ 85us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 11:44:53.836096 (+ 2352us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 11:44:53.836097 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 11:44:53.836098 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 11:44:53.836105 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0620 11:44:53.837834 (+ 1729us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 11:44:53.838265 (+ 431us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 11:44:53.838266 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 11:44:53.838266 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 11:44:53.838325 (+ 59us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 11:44:53.838555 (+ 230us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 11:44:53.838556 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 11:44:53.838556 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 11:44:53.838701 (+ 145us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 11:44:53.838771 (+ 70us) server_negotiation.cc:1035] Waiting for connection context
0620 11:44:53.838974 (+ 203us) server_negotiation.cc:299] Negotiation successful
0620 11:44:53.839057 (+ 83us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":25}
I20260620 11:44:53.841387 20505 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 3497f56952e84c439405a71d196b81b5: Bootstrap starting.
I20260620 11:44:53.842199 20505 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 3497f56952e84c439405a71d196b81b5: Neither blocks nor log segments found. Creating new log.
I20260620 11:44:53.842540 20505 log.cc:824] T 00000000000000000000000000000000 P 3497f56952e84c439405a71d196b81b5: Log is configured to *not* fsync() on all Append() calls
I20260620 11:44:53.843446 20505 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 3497f56952e84c439405a71d196b81b5: No bootstrap required, opened a new log
I20260620 11:44:53.845564 20505 raft_consensus.cc:348] T 00000000000000000000000000000000 P 3497f56952e84c439405a71d196b81b5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "619c5ff12b6949eea40c74e81640db02" member_type: VOTER last_known_addr { host: "127.6.234.254" port: 35005 } } peers { permanent_uuid: "3497f56952e84c439405a71d196b81b5" member_type: VOTER last_known_addr { host: "127.6.234.253" port: 41267 } } peers { permanent_uuid: "03184a22d25c48fdb11b3186802d1e2e" member_type: VOTER last_known_addr { host: "127.6.234.252" port: 38097 } }
I20260620 11:44:53.845741 20505 raft_consensus.cc:374] T 00000000000000000000000000000000 P 3497f56952e84c439405a71d196b81b5 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 11:44:53.845837 20505 raft_consensus.cc:729] T 00000000000000000000000000000000 P 3497f56952e84c439405a71d196b81b5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3497f56952e84c439405a71d196b81b5, State: Initialized, Role: FOLLOWER
I20260620 11:44:53.845999 20505 consensus_queue.cc:260] T 00000000000000000000000000000000 P 3497f56952e84c439405a71d196b81b5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "619c5ff12b6949eea40c74e81640db02" member_type: VOTER last_known_addr { host: "127.6.234.254" port: 35005 } } peers { permanent_uuid: "3497f56952e84c439405a71d196b81b5" member_type: VOTER last_known_addr { host: "127.6.234.253" port: 41267 } } peers { permanent_uuid: "03184a22d25c48fdb11b3186802d1e2e" member_type: VOTER last_known_addr { host: "127.6.234.252" port: 38097 } }
I20260620 11:44:53.846294 20622 sys_catalog.cc:455] T 00000000000000000000000000000000 P 3497f56952e84c439405a71d196b81b5 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "619c5ff12b6949eea40c74e81640db02" member_type: VOTER last_known_addr { host: "127.6.234.254" port: 35005 } } peers { permanent_uuid: "3497f56952e84c439405a71d196b81b5" member_type: VOTER last_known_addr { host: "127.6.234.253" port: 41267 } } peers { permanent_uuid: "03184a22d25c48fdb11b3186802d1e2e" member_type: VOTER last_known_addr { host: "127.6.234.252" port: 38097 } } }
I20260620 11:44:53.846375 20505 sys_catalog.cc:564] T 00000000000000000000000000000000 P 3497f56952e84c439405a71d196b81b5 [sys.catalog]: configured and running, proceeding with master startup.
I20260620 11:44:53.846447 20622 sys_catalog.cc:458] T 00000000000000000000000000000000 P 3497f56952e84c439405a71d196b81b5 [sys.catalog]: This master's current role is: FOLLOWER
W20260620 11:44:53.848315 20633 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 3497f56952e84c439405a71d196b81b5: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260620 11:44:53.848410 20633 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260620 11:44:53.850524 20437 master_runner.cc:263] No leader master found from master 3497f56952e84c439405a71d196b81b5
I20260620 11:44:53.850574 20437 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20260620 11:44:53.863377 20594 raft_consensus.cc:482] T 00000000000000000000000000000000 P 03184a22d25c48fdb11b3186802d1e2e [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260620 11:44:53.863476 20594 raft_consensus.cc:504] T 00000000000000000000000000000000 P 03184a22d25c48fdb11b3186802d1e2e [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "619c5ff12b6949eea40c74e81640db02" member_type: VOTER last_known_addr { host: "127.6.234.254" port: 35005 } } peers { permanent_uuid: "3497f56952e84c439405a71d196b81b5" member_type: VOTER last_known_addr { host: "127.6.234.253" port: 41267 } } peers { permanent_uuid: "03184a22d25c48fdb11b3186802d1e2e" member_type: VOTER last_known_addr { host: "127.6.234.252" port: 38097 } }
I20260620 11:44:53.863677 20594 leader_election.cc:288] T 00000000000000000000000000000000 P 03184a22d25c48fdb11b3186802d1e2e [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 619c5ff12b6949eea40c74e81640db02 (127.6.234.254:35005), 3497f56952e84c439405a71d196b81b5 (127.6.234.253:41267)
I20260620 11:44:53.864094 20404 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "03184a22d25c48fdb11b3186802d1e2e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "619c5ff12b6949eea40c74e81640db02" is_pre_election: true
I20260620 11:44:53.864236 20404 raft_consensus.cc:2454] T 00000000000000000000000000000000 P 619c5ff12b6949eea40c74e81640db02 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 03184a22d25c48fdb11b3186802d1e2e in term 0.
I20260620 11:44:53.864516 20530 leader_election.cc:302] T 00000000000000000000000000000000 P 03184a22d25c48fdb11b3186802d1e2e [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 03184a22d25c48fdb11b3186802d1e2e, 619c5ff12b6949eea40c74e81640db02; no voters:
I20260620 11:44:53.864687 20594 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 03184a22d25c48fdb11b3186802d1e2e [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260620 11:44:53.864733 20594 raft_consensus.cc:482] T 00000000000000000000000000000000 P 03184a22d25c48fdb11b3186802d1e2e [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260620 11:44:53.864755 20594 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 03184a22d25c48fdb11b3186802d1e2e [term 0 FOLLOWER]: Advancing to term 1
I20260620 11:44:53.865561 20594 raft_consensus.cc:504] T 00000000000000000000000000000000 P 03184a22d25c48fdb11b3186802d1e2e [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "619c5ff12b6949eea40c74e81640db02" member_type: VOTER last_known_addr { host: "127.6.234.254" port: 35005 } } peers { permanent_uuid: "3497f56952e84c439405a71d196b81b5" member_type: VOTER last_known_addr { host: "127.6.234.253" port: 41267 } } peers { permanent_uuid: "03184a22d25c48fdb11b3186802d1e2e" member_type: VOTER last_known_addr { host: "127.6.234.252" port: 38097 } }
I20260620 11:44:53.865998 20404 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "03184a22d25c48fdb11b3186802d1e2e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "619c5ff12b6949eea40c74e81640db02"
I20260620 11:44:53.866067 20404 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 619c5ff12b6949eea40c74e81640db02 [term 0 FOLLOWER]: Advancing to term 1
I20260620 11:44:53.866807 20404 raft_consensus.cc:2454] T 00000000000000000000000000000000 P 619c5ff12b6949eea40c74e81640db02 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 03184a22d25c48fdb11b3186802d1e2e in term 1.
I20260620 11:44:53.867605 20479 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "03184a22d25c48fdb11b3186802d1e2e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3497f56952e84c439405a71d196b81b5" is_pre_election: true
I20260620 11:44:53.867760 20479 raft_consensus.cc:2454] T 00000000000000000000000000000000 P 3497f56952e84c439405a71d196b81b5 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 03184a22d25c48fdb11b3186802d1e2e in term 0.
I20260620 11:44:53.865669 20594 leader_election.cc:288] T 00000000000000000000000000000000 P 03184a22d25c48fdb11b3186802d1e2e [CANDIDATE]: Term 1 election: Requested vote from peers 619c5ff12b6949eea40c74e81640db02 (127.6.234.254:35005), 3497f56952e84c439405a71d196b81b5 (127.6.234.253:41267)
I20260620 11:44:53.867353 20530 leader_election.cc:302] T 00000000000000000000000000000000 P 03184a22d25c48fdb11b3186802d1e2e [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 03184a22d25c48fdb11b3186802d1e2e, 619c5ff12b6949eea40c74e81640db02; no voters:
I20260620 11:44:53.867607 20478 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "03184a22d25c48fdb11b3186802d1e2e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3497f56952e84c439405a71d196b81b5"
I20260620 11:44:53.867868 20478 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 3497f56952e84c439405a71d196b81b5 [term 0 FOLLOWER]: Advancing to term 1
I20260620 11:44:53.868037 20594 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 03184a22d25c48fdb11b3186802d1e2e [term 1 FOLLOWER]: Leader election won for term 1
I20260620 11:44:53.868186 20594 raft_consensus.cc:686] T 00000000000000000000000000000000 P 03184a22d25c48fdb11b3186802d1e2e [term 1 LEADER]: Becoming Leader. State: Replica: 03184a22d25c48fdb11b3186802d1e2e, State: Running, Role: LEADER
I20260620 11:44:53.868268 20594 consensus_queue.cc:237] T 00000000000000000000000000000000 P 03184a22d25c48fdb11b3186802d1e2e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "619c5ff12b6949eea40c74e81640db02" member_type: VOTER last_known_addr { host: "127.6.234.254" port: 35005 } } peers { permanent_uuid: "3497f56952e84c439405a71d196b81b5" member_type: VOTER last_known_addr { host: "127.6.234.253" port: 41267 } } peers { permanent_uuid: "03184a22d25c48fdb11b3186802d1e2e" member_type: VOTER last_known_addr { host: "127.6.234.252" port: 38097 } }
I20260620 11:44:53.868646 20594 sys_catalog.cc:455] T 00000000000000000000000000000000 P 03184a22d25c48fdb11b3186802d1e2e [sys.catalog]: SysCatalogTable state changed. Reason: New leader 03184a22d25c48fdb11b3186802d1e2e. Latest consensus state: current_term: 1 leader_uuid: "03184a22d25c48fdb11b3186802d1e2e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "619c5ff12b6949eea40c74e81640db02" member_type: VOTER last_known_addr { host: "127.6.234.254" port: 35005 } } peers { permanent_uuid: "3497f56952e84c439405a71d196b81b5" member_type: VOTER last_known_addr { host: "127.6.234.253" port: 41267 } } peers { permanent_uuid: "03184a22d25c48fdb11b3186802d1e2e" member_type: VOTER last_known_addr { host: "127.6.234.252" port: 38097 } } }
I20260620 11:44:53.868710 20594 sys_catalog.cc:458] T 00000000000000000000000000000000 P 03184a22d25c48fdb11b3186802d1e2e [sys.catalog]: This master's current role is: LEADER
I20260620 11:44:53.868670 20478 raft_consensus.cc:2454] T 00000000000000000000000000000000 P 3497f56952e84c439405a71d196b81b5 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 03184a22d25c48fdb11b3186802d1e2e in term 1.
I20260620 11:44:53.868908 20637 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260620 11:44:53.869138 20637 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260620 11:44:53.871066 20478 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 3497f56952e84c439405a71d196b81b5 [term 1 FOLLOWER]: Refusing update from remote peer 03184a22d25c48fdb11b3186802d1e2e: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 11:44:53.871095 20404 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 619c5ff12b6949eea40c74e81640db02 [term 1 FOLLOWER]: Refusing update from remote peer 03184a22d25c48fdb11b3186802d1e2e: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 11:44:53.871388 20594 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 03184a22d25c48fdb11b3186802d1e2e [LEADER]: Connected to new peer: Peer: permanent_uuid: "619c5ff12b6949eea40c74e81640db02" member_type: VOTER last_known_addr { host: "127.6.234.254" port: 35005 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 11:44:53.871529 20594 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 03184a22d25c48fdb11b3186802d1e2e [LEADER]: Connected to new peer: Peer: permanent_uuid: "3497f56952e84c439405a71d196b81b5" member_type: VOTER last_known_addr { host: "127.6.234.253" port: 41267 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 11:44:53.872756 20622 sys_catalog.cc:455] T 00000000000000000000000000000000 P 3497f56952e84c439405a71d196b81b5 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 03184a22d25c48fdb11b3186802d1e2e. Latest consensus state: current_term: 1 leader_uuid: "03184a22d25c48fdb11b3186802d1e2e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "619c5ff12b6949eea40c74e81640db02" member_type: VOTER last_known_addr { host: "127.6.234.254" port: 35005 } } peers { permanent_uuid: "3497f56952e84c439405a71d196b81b5" member_type: VOTER last_known_addr { host: "127.6.234.253" port: 41267 } } peers { permanent_uuid: "03184a22d25c48fdb11b3186802d1e2e" member_type: VOTER last_known_addr { host: "127.6.234.252" port: 38097 } } }
I20260620 11:44:53.872848 20622 sys_catalog.cc:458] T 00000000000000000000000000000000 P 3497f56952e84c439405a71d196b81b5 [sys.catalog]: This master's current role is: FOLLOWER
I20260620 11:44:53.873925 20608 sys_catalog.cc:455] T 00000000000000000000000000000000 P 619c5ff12b6949eea40c74e81640db02 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 03184a22d25c48fdb11b3186802d1e2e. Latest consensus state: current_term: 1 leader_uuid: "03184a22d25c48fdb11b3186802d1e2e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "619c5ff12b6949eea40c74e81640db02" member_type: VOTER last_known_addr { host: "127.6.234.254" port: 35005 } } peers { permanent_uuid: "3497f56952e84c439405a71d196b81b5" member_type: VOTER last_known_addr { host: "127.6.234.253" port: 41267 } } peers { permanent_uuid: "03184a22d25c48fdb11b3186802d1e2e" member_type: VOTER last_known_addr { host: "127.6.234.252" port: 38097 } } }
I20260620 11:44:53.874002 20608 sys_catalog.cc:458] T 00000000000000000000000000000000 P 619c5ff12b6949eea40c74e81640db02 [sys.catalog]: This master's current role is: FOLLOWER
I20260620 11:44:53.875206 20635 sys_catalog.cc:455] T 00000000000000000000000000000000 P 03184a22d25c48fdb11b3186802d1e2e [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "03184a22d25c48fdb11b3186802d1e2e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "619c5ff12b6949eea40c74e81640db02" member_type: VOTER last_known_addr { host: "127.6.234.254" port: 35005 } } peers { permanent_uuid: "3497f56952e84c439405a71d196b81b5" member_type: VOTER last_known_addr { host: "127.6.234.253" port: 41267 } } peers { permanent_uuid: "03184a22d25c48fdb11b3186802d1e2e" member_type: VOTER last_known_addr { host: "127.6.234.252" port: 38097 } } }
I20260620 11:44:53.875877 20622 sys_catalog.cc:455] T 00000000000000000000000000000000 P 3497f56952e84c439405a71d196b81b5 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "03184a22d25c48fdb11b3186802d1e2e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "619c5ff12b6949eea40c74e81640db02" member_type: VOTER last_known_addr { host: "127.6.234.254" port: 35005 } } peers { permanent_uuid: "3497f56952e84c439405a71d196b81b5" member_type: VOTER last_known_addr { host: "127.6.234.253" port: 41267 } } peers { permanent_uuid: "03184a22d25c48fdb11b3186802d1e2e" member_type: VOTER last_known_addr { host: "127.6.234.252" port: 38097 } } }
I20260620 11:44:53.875952 20622 sys_catalog.cc:458] T 00000000000000000000000000000000 P 3497f56952e84c439405a71d196b81b5 [sys.catalog]: This master's current role is: FOLLOWER
I20260620 11:44:53.876026 20635 sys_catalog.cc:458] T 00000000000000000000000000000000 P 03184a22d25c48fdb11b3186802d1e2e [sys.catalog]: This master's current role is: LEADER
I20260620 11:44:53.876260 20638 mvcc.cc:205] Tried to move back new op lower bound from 7298891341293699072 to 7298891341285064704. Current Snapshot: MvccSnapshot[applied={T|T < 7298891341293699072}]
I20260620 11:44:53.876437 20640 mvcc.cc:205] Tried to move back new op lower bound from 7298891341293699072 to 7298891341285064704. Current Snapshot: MvccSnapshot[applied={T|T < 7298891341293699072}]
I20260620 11:44:53.877478 20635 sys_catalog.cc:455] T 00000000000000000000000000000000 P 03184a22d25c48fdb11b3186802d1e2e [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "03184a22d25c48fdb11b3186802d1e2e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "619c5ff12b6949eea40c74e81640db02" member_type: VOTER last_known_addr { host: "127.6.234.254" port: 35005 } } peers { permanent_uuid: "3497f56952e84c439405a71d196b81b5" member_type: VOTER last_known_addr { host: "127.6.234.253" port: 41267 } } peers { permanent_uuid: "03184a22d25c48fdb11b3186802d1e2e" member_type: VOTER last_known_addr { host: "127.6.234.252" port: 38097 } } }
I20260620 11:44:53.877552 20635 sys_catalog.cc:458] T 00000000000000000000000000000000 P 03184a22d25c48fdb11b3186802d1e2e [sys.catalog]: This master's current role is: LEADER
I20260620 11:44:53.878055 20608 sys_catalog.cc:455] T 00000000000000000000000000000000 P 619c5ff12b6949eea40c74e81640db02 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "03184a22d25c48fdb11b3186802d1e2e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "619c5ff12b6949eea40c74e81640db02" member_type: VOTER last_known_addr { host: "127.6.234.254" port: 35005 } } peers { permanent_uuid: "3497f56952e84c439405a71d196b81b5" member_type: VOTER last_known_addr { host: "127.6.234.253" port: 41267 } } peers { permanent_uuid: "03184a22d25c48fdb11b3186802d1e2e" member_type: VOTER last_known_addr { host: "127.6.234.252" port: 38097 } } }
I20260620 11:44:53.878120 20637 catalog_manager.cc:1347] Generated new cluster ID: fb63832845a740daae5c05a0ce1f0860
I20260620 11:44:53.878152 20608 sys_catalog.cc:458] T 00000000000000000000000000000000 P 619c5ff12b6949eea40c74e81640db02 [sys.catalog]: This master's current role is: FOLLOWER
I20260620 11:44:53.878156 20637 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260620 11:44:53.878494 20642 mvcc.cc:205] Tried to move back new op lower bound from 7298891341293699072 to 7298891341285064704. Current Snapshot: MvccSnapshot[applied={T|T < 7298891341293699072}]
I20260620 11:44:53.899439 20637 catalog_manager.cc:1370] Generated new certificate authority record
I20260620 11:44:53.900279 20637 catalog_manager.cc:1504] Loading token signing keys...
I20260620 11:44:53.908456 20637 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 03184a22d25c48fdb11b3186802d1e2e: Generated new TSK 0
I20260620 11:44:53.908711 20637 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260620 11:44:54.800971 20619 catalog_manager.cc:1259] Loaded cluster ID: fb63832845a740daae5c05a0ce1f0860
I20260620 11:44:54.801091 20619 catalog_manager.cc:1552] T 00000000000000000000000000000000 P 619c5ff12b6949eea40c74e81640db02: loading cluster ID for follower catalog manager: success
I20260620 11:44:54.802000 20619 catalog_manager.cc:1574] T 00000000000000000000000000000000 P 619c5ff12b6949eea40c74e81640db02: acquiring CA information for follower catalog manager: success
I20260620 11:44:54.802263 20619 catalog_manager.cc:1602] T 00000000000000000000000000000000 P 619c5ff12b6949eea40c74e81640db02: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20260620 11:44:54.848862 20633 catalog_manager.cc:1259] Loaded cluster ID: fb63832845a740daae5c05a0ce1f0860
I20260620 11:44:54.848938 20633 catalog_manager.cc:1552] T 00000000000000000000000000000000 P 3497f56952e84c439405a71d196b81b5: loading cluster ID for follower catalog manager: success
I20260620 11:44:54.849787 20633 catalog_manager.cc:1574] T 00000000000000000000000000000000 P 3497f56952e84c439405a71d196b81b5: acquiring CA information for follower catalog manager: success
I20260620 11:44:54.850122 20633 catalog_manager.cc:1602] T 00000000000000000000000000000000 P 3497f56952e84c439405a71d196b81b5: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20260620 11:44:54.975124 7083 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task1JRg7N/build/release/bin/kudu with pid 20362
I20260620 11:44:54.980373 7083 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task1JRg7N/build/release/bin/kudu with pid 20437
I20260620 11:44:54.985172 7083 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task1JRg7N/build/release/bin/kudu with pid 20512
2026-06-20T11:44:54Z chronyd exiting
[ OK ] SecurityITest.IPKICACert (2829 ms)
[ RUN ] SecurityITest.IPKICACertWebServerEndPoints
Loading random data
Initializing database '/tmp/dist-test-task1JRg7N/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1781955746769051-7083-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 20 11:44:55 dist-test-slave-cjc6 krb5kdc[20654](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 20 11:44:55 dist-test-slave-cjc6 krb5kdc[20654](info): set up 2 sockets
Jun 20 11:44:55 dist-test-slave-cjc6 krb5kdc[20654](info): commencing operation
krb5kdc: starting...
W20260620 11:44:56.021884 7083 mini_kdc.cc:121] Time spent starting KDC: real 1.017s user 0.000s sys 0.006s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task1JRg7N/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1781955746769051-7083-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task1JRg7N/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1781955746769051-7083-0/krb5kdc/test-user.keytab.
Jun 20 11:44:56 dist-test-slave-cjc6 krb5kdc[20654](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781955896, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-20T11:44:56Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-20T11:44:56Z Disabled control of system clock
WARNING: no policy specified for kudu/127.6.234.254@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.6.234.254@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.6.234.254 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task1JRg7N/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1781955746769051-7083-0/krb5kdc/kudu_127.6.234.254.keytab.
Entry for principal kudu/127.6.234.254 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task1JRg7N/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1781955746769051-7083-0/krb5kdc/kudu_127.6.234.254.keytab.
I20260620 11:44:56.140559 7083 external_mini_cluster.cc:1326] Running /tmp/dist-test-task1JRg7N/build/release/bin/kudu
/tmp/dist-test-task1JRg7N/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task1JRg7N/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1781955746769051-7083-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task1JRg7N/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1781955746769051-7083-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task1JRg7N/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1781955746769051-7083-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task1JRg7N/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1781955746769051-7083-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.6.234.254:36067
--webserver_interface=127.6.234.254
--webserver_port=0
--keytab_file=/tmp/dist-test-task1JRg7N/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1781955746769051-7083-0/krb5kdc/kudu_127.6.234.254.keytab
--principal=kudu/127.6.234.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.6.234.212:33455
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.6.234.254:36067
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task1JRg7N/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1781955746769051-7083-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task1JRg7N/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1781955746769051-7083-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task1JRg7N/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1781955746769051-7083-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 11:44:56.220448 20667 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 11:44:56.220687 20667 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 11:44:56.220736 20667 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 11:44:56.222009 20667 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 11:44:56.222071 20667 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 11:44:56.222095 20667 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 11:44:56.222117 20667 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 11:44:56.222147 20667 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 11:44:56.223593 20667 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.6.234.212:33455
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task1JRg7N/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1781955746769051-7083-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task1JRg7N/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1781955746769051-7083-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.6.234.254:36067
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task1JRg7N/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1781955746769051-7083-0/krb5kdc/kudu_127.6.234.254.keytab
--principal=kudu/127.6.234.254
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.6.234.254:36067
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task1JRg7N/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1781955746769051-7083-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.6.234.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.20667
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task1JRg7N/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1781955746769051-7083-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 20 Jun 2026 10:32:37 UTC on ccycloud.quasar-bgyofv.root.comops.site
build id 4457173
I20260620 11:44:56.223874 20667 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 11:44:56.224098 20667 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 11:44:56.228820 20673 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 11:44:56.228832 20672 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 11:44:56.228888 20675 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 11:44:56.229023 20667 server_base.cc:1034] running on GCE node
I20260620 11:44:56.229351 20667 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 11:44:56.229640 20667 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 11:44:56.230813 20667 hybrid_clock.cc:648] HybridClock initialized: now 1781955896230792 us; error 39 us; skew 500 ppm
Jun 20 11:44:56 dist-test-slave-cjc6 krb5kdc[20654](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781955896, etypes {rep=17 tkt=17 ses=17}, kudu/127.6.234.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 11:44:56.233188 20667 init.cc:377] Logged in from keytab as kudu/127.6.234.254@KRBTEST.COM (short username kudu)
I20260620 11:44:56.233610 20667 webserver.cc:460] Webserver started at http://127.6.234.254:43007/ using document root <none> and password file <none>
I20260620 11:44:56.233856 20667 fs_manager.cc:359] Metadata directory not provided
I20260620 11:44:56.233927 20667 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 11:44:56.234076 20667 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 11:44:56.235162 20667 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task1JRg7N/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1781955746769051-7083-0/minicluster-data/master-0/data/instance:
uuid: "811690a8e9104d3bbed58a6e8143093e"
format_stamp: "Formatted at 2026-06-20 11:44:56 on dist-test-slave-cjc6"
server_key: "35c6f6fc4b2bd0eb6b626c27b29554b9"
server_key_iv: "301e22a70c4c52532fab30d03ef5cb5e"
server_key_version: "encryptionkey@0"
I20260620 11:44:56.235517 20667 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task1JRg7N/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1781955746769051-7083-0/minicluster-data/master-0/wal/instance:
uuid: "811690a8e9104d3bbed58a6e8143093e"
format_stamp: "Formatted at 2026-06-20 11:44:56 on dist-test-slave-cjc6"
server_key: "35c6f6fc4b2bd0eb6b626c27b29554b9"
server_key_iv: "301e22a70c4c52532fab30d03ef5cb5e"
server_key_version: "encryptionkey@0"
I20260620 11:44:56.236794 20667 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.001s
I20260620 11:44:56.237496 20682 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 11:44:56.237679 20667 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.000s
I20260620 11:44:56.237818 20667 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task1JRg7N/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1781955746769051-7083-0/minicluster-data/master-0/data,/tmp/dist-test-task1JRg7N/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1781955746769051-7083-0/minicluster-data/master-0/wal
uuid: "811690a8e9104d3bbed58a6e8143093e"
format_stamp: "Formatted at 2026-06-20 11:44:56 on dist-test-slave-cjc6"
server_key: "35c6f6fc4b2bd0eb6b626c27b29554b9"
server_key_iv: "301e22a70c4c52532fab30d03ef5cb5e"
server_key_version: "encryptionkey@0"
I20260620 11:44:56.237903 20667 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task1JRg7N/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1781955746769051-7083-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task1JRg7N/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1781955746769051-7083-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task1JRg7N/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1781955746769051-7083-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 11:44:56.249595 20667 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 11:44:56.251402 20667 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 11:44:56.251569 20667 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 11:44:56.255470 20667 rpc_server.cc:307] RPC server started. Bound to: 127.6.234.254:36067
I20260620 11:44:56.255502 20734 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.6.234.254:36067 every 8 connection(s)
I20260620 11:44:56.255839 20667 server_base.cc:1166] Dumped server information to /tmp/dist-test-task1JRg7N/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1781955746769051-7083-0/minicluster-data/master-0/data/info.pb
I20260620 11:44:56.256711 20735 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 11:44:56.259080 20735 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 811690a8e9104d3bbed58a6e8143093e: Bootstrap starting.
I20260620 11:44:56.259660 20735 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 811690a8e9104d3bbed58a6e8143093e: Neither blocks nor log segments found. Creating new log.
I20260620 11:44:56.259946 20735 log.cc:824] T 00000000000000000000000000000000 P 811690a8e9104d3bbed58a6e8143093e: Log is configured to *not* fsync() on all Append() calls
I20260620 11:44:56.260607 20735 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 811690a8e9104d3bbed58a6e8143093e: No bootstrap required, opened a new log
I20260620 11:44:56.262245 20735 raft_consensus.cc:348] T 00000000000000000000000000000000 P 811690a8e9104d3bbed58a6e8143093e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "811690a8e9104d3bbed58a6e8143093e" member_type: VOTER last_known_addr { host: "127.6.234.254" port: 36067 } }
I20260620 11:44:56.262398 20735 raft_consensus.cc:374] T 00000000000000000000000000000000 P 811690a8e9104d3bbed58a6e8143093e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 11:44:56.262427 20735 raft_consensus.cc:729] T 00000000000000000000000000000000 P 811690a8e9104d3bbed58a6e8143093e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 811690a8e9104d3bbed58a6e8143093e, State: Initialized, Role: FOLLOWER
I20260620 11:44:56.262598 20735 consensus_queue.cc:260] T 00000000000000000000000000000000 P 811690a8e9104d3bbed58a6e8143093e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "811690a8e9104d3bbed58a6e8143093e" member_type: VOTER last_known_addr { host: "127.6.234.254" port: 36067 } }
I20260620 11:44:56.262660 20735 raft_consensus.cc:388] T 00000000000000000000000000000000 P 811690a8e9104d3bbed58a6e8143093e [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260620 11:44:56.262681 20735 raft_consensus.cc:482] T 00000000000000000000000000000000 P 811690a8e9104d3bbed58a6e8143093e [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260620 11:44:56.262743 20735 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 811690a8e9104d3bbed58a6e8143093e [term 0 FOLLOWER]: Advancing to term 1
I20260620 11:44:56.263299 20735 raft_consensus.cc:504] T 00000000000000000000000000000000 P 811690a8e9104d3bbed58a6e8143093e [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "811690a8e9104d3bbed58a6e8143093e" member_type: VOTER last_known_addr { host: "127.6.234.254" port: 36067 } }
I20260620 11:44:56.263396 20735 leader_election.cc:302] T 00000000000000000000000000000000 P 811690a8e9104d3bbed58a6e8143093e [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 811690a8e9104d3bbed58a6e8143093e; no voters:
I20260620 11:44:56.263599 20735 leader_election.cc:288] T 00000000000000000000000000000000 P 811690a8e9104d3bbed58a6e8143093e [CANDIDATE]: Term 1 election: Requested vote from peers
I20260620 11:44:56.263655 20738 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 811690a8e9104d3bbed58a6e8143093e [term 1 FOLLOWER]: Leader election won for term 1
I20260620 11:44:56.263819 20738 raft_consensus.cc:686] T 00000000000000000000000000000000 P 811690a8e9104d3bbed58a6e8143093e [term 1 LEADER]: Becoming Leader. State: Replica: 811690a8e9104d3bbed58a6e8143093e, State: Running, Role: LEADER
I20260620 11:44:56.263878 20735 sys_catalog.cc:564] T 00000000000000000000000000000000 P 811690a8e9104d3bbed58a6e8143093e [sys.catalog]: configured and running, proceeding with master startup.
I20260620 11:44:56.263943 20738 consensus_queue.cc:237] T 00000000000000000000000000000000 P 811690a8e9104d3bbed58a6e8143093e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "811690a8e9104d3bbed58a6e8143093e" member_type: VOTER last_known_addr { host: "127.6.234.254" port: 36067 } }
I20260620 11:44:56.264305 20740 sys_catalog.cc:455] T 00000000000000000000000000000000 P 811690a8e9104d3bbed58a6e8143093e [sys.catalog]: SysCatalogTable state changed. Reason: New leader 811690a8e9104d3bbed58a6e8143093e. Latest consensus state: current_term: 1 leader_uuid: "811690a8e9104d3bbed58a6e8143093e" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "811690a8e9104d3bbed58a6e8143093e" member_type: VOTER last_known_addr { host: "127.6.234.254" port: 36067 } } }
I20260620 11:44:56.264387 20740 sys_catalog.cc:458] T 00000000000000000000000000000000 P 811690a8e9104d3bbed58a6e8143093e [sys.catalog]: This master's current role is: LEADER
I20260620 11:44:56.264449 20739 sys_catalog.cc:455] T 00000000000000000000000000000000 P 811690a8e9104d3bbed58a6e8143093e [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "811690a8e9104d3bbed58a6e8143093e" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "811690a8e9104d3bbed58a6e8143093e" member_type: VOTER last_known_addr { host: "127.6.234.254" port: 36067 } } }
I20260620 11:44:56.264516 20739 sys_catalog.cc:458] T 00000000000000000000000000000000 P 811690a8e9104d3bbed58a6e8143093e [sys.catalog]: This master's current role is: LEADER
I20260620 11:44:56.264633 20744 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260620 11:44:56.265059 7083 external_mini_cluster.cc:1388] Started /tmp/dist-test-task1JRg7N/build/release/bin/kudu as pid 20667
I20260620 11:44:56.265165 7083 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task1JRg7N/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1781955746769051-7083-0/minicluster-data/master-0/wal/instance
I20260620 11:44:56.265223 20744 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260620 11:44:56.265376 7083 external_mini_cluster.cc:1428] Setting key 1fecdcd66101fac14148460d98bf7e93
I20260620 11:44:56.267244 20744 catalog_manager.cc:1347] Generated new cluster ID: 0495d56654284f8a9a9c6fc21b789624
I20260620 11:44:56.267310 20744 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
Jun 20 11:44:56 dist-test-slave-cjc6 krb5kdc[20654](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781955896, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.6.234.254@KRBTEST.COM
I20260620 11:44:56.277011 20757 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 11:44:56.265947 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49084 (local address 127.6.234.254:36067)
0620 11:44:56.266158 (+ 211us) server_negotiation.cc:206] Beginning negotiation
0620 11:44:56.266168 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0620 11:44:56.266203 (+ 35us) server_negotiation.cc:407] Connection header received
0620 11:44:56.266319 (+ 116us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 11:44:56.266335 (+ 16us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 11:44:56.266408 (+ 73us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 11:44:56.266548 (+ 140us) server_negotiation.cc:226] Negotiated authn=SASL
0620 11:44:56.267250 (+ 702us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 11:44:56.268693 (+ 1443us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 11:44:56.269749 (+ 1056us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 11:44:56.270404 (+ 655us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 11:44:56.270518 (+ 114us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 11:44:56.273308 (+ 2790us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 11:44:56.273312 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 11:44:56.273312 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 11:44:56.273324 (+ 12us) server_negotiation.cc:922] calling sasl_server_start()
0620 11:44:56.275319 (+ 1995us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 11:44:56.275741 (+ 422us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 11:44:56.275744 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 11:44:56.275744 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 11:44:56.275854 (+ 110us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 11:44:56.276066 (+ 212us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 11:44:56.276067 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 11:44:56.276067 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 11:44:56.276363 (+ 296us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 11:44:56.276510 (+ 147us) server_negotiation.cc:1035] Waiting for connection context
0620 11:44:56.276633 (+ 123us) server_negotiation.cc:299] Negotiation successful
0620 11:44:56.276764 (+ 131us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":163,"thread_start_us":125,"threads_started":1}
I20260620 11:44:56.277846 20744 catalog_manager.cc:1370] Generated new certificate authority record
I20260620 11:44:56.278074 7083 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260620 11:44:56.278458 20744 catalog_manager.cc:1504] Loading token signing keys...
I20260620 11:44:56.289084 20744 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 811690a8e9104d3bbed58a6e8143093e: Generated new TSK 0
I20260620 11:44:56.289283 20744 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260620 11:44:56.332237 7083 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task1JRg7N/build/release/bin/kudu with pid 20667
2026-06-20T11:44:56Z chronyd exiting
[ OK ] SecurityITest.IPKICACertWebServerEndPoints (1345 ms)
[ RUN ] SecurityITest.TestEnableAndDisableEncryptedIPKICert
Loading random data
Initializing database '/tmp/dist-test-task1JRg7N/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781955746769051-7083-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 20 11:44:56 dist-test-slave-cjc6 krb5kdc[20762](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 20 11:44:56 dist-test-slave-cjc6 krb5kdc[20762](info): set up 2 sockets
Jun 20 11:44:56 dist-test-slave-cjc6 krb5kdc[20762](info): commencing operation
krb5kdc: starting...
W20260620 11:44:57.369782 7083 mini_kdc.cc:121] Time spent starting KDC: real 1.019s user 0.003s sys 0.003s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task1JRg7N/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781955746769051-7083-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task1JRg7N/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781955746769051-7083-0/krb5kdc/test-user.keytab.
Jun 20 11:44:57 dist-test-slave-cjc6 krb5kdc[20762](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781955897, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-20T11:44:57Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-20T11:44:57Z Disabled control of system clock
WARNING: no policy specified for kudu/127.6.234.254@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.6.234.254@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.6.234.254 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task1JRg7N/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781955746769051-7083-0/krb5kdc/kudu_127.6.234.254.keytab.
Entry for principal kudu/127.6.234.254 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task1JRg7N/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781955746769051-7083-0/krb5kdc/kudu_127.6.234.254.keytab.
I20260620 11:44:57.483734 7083 external_mini_cluster.cc:1326] Running /tmp/dist-test-task1JRg7N/build/release/bin/kudu
/tmp/dist-test-task1JRg7N/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task1JRg7N/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781955746769051-7083-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task1JRg7N/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781955746769051-7083-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task1JRg7N/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781955746769051-7083-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task1JRg7N/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781955746769051-7083-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.6.234.254:44633
--webserver_interface=127.6.234.254
--webserver_port=0
--keytab_file=/tmp/dist-test-task1JRg7N/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781955746769051-7083-0/krb5kdc/kudu_127.6.234.254.keytab
--principal=kudu/127.6.234.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.6.234.212:34463
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.6.234.254:44633
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task1JRg7N/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781955746769051-7083-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task1JRg7N/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781955746769051-7083-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task1JRg7N/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781955746769051-7083-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 11:44:57.562306 20775 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 11:44:57.562507 20775 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 11:44:57.562529 20775 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 11:44:57.563714 20775 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 11:44:57.563756 20775 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 11:44:57.563771 20775 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 11:44:57.563783 20775 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 11:44:57.563843 20775 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 11:44:57.565367 20775 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.6.234.212:34463
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task1JRg7N/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781955746769051-7083-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task1JRg7N/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781955746769051-7083-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.6.234.254:44633
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task1JRg7N/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781955746769051-7083-0/krb5kdc/kudu_127.6.234.254.keytab
--principal=kudu/127.6.234.254
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.6.234.254:44633
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task1JRg7N/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781955746769051-7083-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.6.234.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.20775
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task1JRg7N/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781955746769051-7083-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 20 Jun 2026 10:32:37 UTC on ccycloud.quasar-bgyofv.root.comops.site
build id 4457173
I20260620 11:44:57.565649 20775 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 11:44:57.565904 20775 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 11:44:57.570737 20783 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 11:44:57.570909 20775 server_base.cc:1034] running on GCE node
W20260620 11:44:57.570853 20780 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 11:44:57.570964 20781 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 11:44:57.571295 20775 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 11:44:57.571584 20775 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 11:44:57.572747 20775 hybrid_clock.cc:648] HybridClock initialized: now 1781955897572730 us; error 61 us; skew 500 ppm
Jun 20 11:44:57 dist-test-slave-cjc6 krb5kdc[20762](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781955897, etypes {rep=17 tkt=17 ses=17}, kudu/127.6.234.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 11:44:57.575399 20775 init.cc:377] Logged in from keytab as kudu/127.6.234.254@KRBTEST.COM (short username kudu)
I20260620 11:44:57.575822 20775 webserver.cc:460] Webserver started at http://127.6.234.254:39859/ using document root <none> and password file <none>
I20260620 11:44:57.576056 20775 fs_manager.cc:359] Metadata directory not provided
I20260620 11:44:57.576151 20775 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 11:44:57.576301 20775 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 11:44:57.577334 20775 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task1JRg7N/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781955746769051-7083-0/minicluster-data/master-0/data/instance:
uuid: "4195d863c1404c0a8137a88b6d8ad6f1"
format_stamp: "Formatted at 2026-06-20 11:44:57 on dist-test-slave-cjc6"
server_key: "6bc565ce9f35dadb384c5433eecd9a63"
server_key_iv: "9b7b41592b4da7fed09776909d1dedc9"
server_key_version: "encryptionkey@0"
I20260620 11:44:57.577762 20775 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task1JRg7N/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781955746769051-7083-0/minicluster-data/master-0/wal/instance:
uuid: "4195d863c1404c0a8137a88b6d8ad6f1"
format_stamp: "Formatted at 2026-06-20 11:44:57 on dist-test-slave-cjc6"
server_key: "6bc565ce9f35dadb384c5433eecd9a63"
server_key_iv: "9b7b41592b4da7fed09776909d1dedc9"
server_key_version: "encryptionkey@0"
I20260620 11:44:57.579047 20775 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.001s
I20260620 11:44:57.579823 20790 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 11:44:57.580014 20775 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.000s
I20260620 11:44:57.580114 20775 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task1JRg7N/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781955746769051-7083-0/minicluster-data/master-0/data,/tmp/dist-test-task1JRg7N/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781955746769051-7083-0/minicluster-data/master-0/wal
uuid: "4195d863c1404c0a8137a88b6d8ad6f1"
format_stamp: "Formatted at 2026-06-20 11:44:57 on dist-test-slave-cjc6"
server_key: "6bc565ce9f35dadb384c5433eecd9a63"
server_key_iv: "9b7b41592b4da7fed09776909d1dedc9"
server_key_version: "encryptionkey@0"
I20260620 11:44:57.580196 20775 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task1JRg7N/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781955746769051-7083-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task1JRg7N/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781955746769051-7083-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task1JRg7N/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781955746769051-7083-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 11:44:57.596442 20775 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 11:44:57.598479 20775 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 11:44:57.598641 20775 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 11:44:57.602442 20775 rpc_server.cc:307] RPC server started. Bound to: 127.6.234.254:44633
I20260620 11:44:57.602494 20842 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.6.234.254:44633 every 8 connection(s)
I20260620 11:44:57.602854 20775 server_base.cc:1166] Dumped server information to /tmp/dist-test-task1JRg7N/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781955746769051-7083-0/minicluster-data/master-0/data/info.pb
I20260620 11:44:57.603818 20843 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 11:44:57.606199 20843 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 4195d863c1404c0a8137a88b6d8ad6f1: Bootstrap starting.
I20260620 11:44:57.606777 20843 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 4195d863c1404c0a8137a88b6d8ad6f1: Neither blocks nor log segments found. Creating new log.
I20260620 11:44:57.607056 20843 log.cc:824] T 00000000000000000000000000000000 P 4195d863c1404c0a8137a88b6d8ad6f1: Log is configured to *not* fsync() on all Append() calls
I20260620 11:44:57.607733 20843 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 4195d863c1404c0a8137a88b6d8ad6f1: No bootstrap required, opened a new log
I20260620 11:44:57.608286 7083 external_mini_cluster.cc:1388] Started /tmp/dist-test-task1JRg7N/build/release/bin/kudu as pid 20775
I20260620 11:44:57.608417 7083 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task1JRg7N/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781955746769051-7083-0/minicluster-data/master-0/wal/instance
I20260620 11:44:57.608614 7083 external_mini_cluster.cc:1428] Setting key 41ef4fe4b51ff0f112667e19c4e7b049
I20260620 11:44:57.610435 20843 raft_consensus.cc:348] T 00000000000000000000000000000000 P 4195d863c1404c0a8137a88b6d8ad6f1 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4195d863c1404c0a8137a88b6d8ad6f1" member_type: VOTER last_known_addr { host: "127.6.234.254" port: 44633 } }
I20260620 11:44:57.610632 20843 raft_consensus.cc:374] T 00000000000000000000000000000000 P 4195d863c1404c0a8137a88b6d8ad6f1 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 11:44:57.610694 20843 raft_consensus.cc:729] T 00000000000000000000000000000000 P 4195d863c1404c0a8137a88b6d8ad6f1 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4195d863c1404c0a8137a88b6d8ad6f1, State: Initialized, Role: FOLLOWER
I20260620 11:44:57.610831 20843 consensus_queue.cc:260] T 00000000000000000000000000000000 P 4195d863c1404c0a8137a88b6d8ad6f1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4195d863c1404c0a8137a88b6d8ad6f1" member_type: VOTER last_known_addr { host: "127.6.234.254" port: 44633 } }
I20260620 11:44:57.610932 20843 raft_consensus.cc:388] T 00000000000000000000000000000000 P 4195d863c1404c0a8137a88b6d8ad6f1 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260620 11:44:57.610975 20843 raft_consensus.cc:482] T 00000000000000000000000000000000 P 4195d863c1404c0a8137a88b6d8ad6f1 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260620 11:44:57.611023 20843 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 4195d863c1404c0a8137a88b6d8ad6f1 [term 0 FOLLOWER]: Advancing to term 1
I20260620 11:44:57.611532 20843 raft_consensus.cc:504] T 00000000000000000000000000000000 P 4195d863c1404c0a8137a88b6d8ad6f1 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4195d863c1404c0a8137a88b6d8ad6f1" member_type: VOTER last_known_addr { host: "127.6.234.254" port: 44633 } }
I20260620 11:44:57.611651 20843 leader_election.cc:302] T 00000000000000000000000000000000 P 4195d863c1404c0a8137a88b6d8ad6f1 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 4195d863c1404c0a8137a88b6d8ad6f1; no voters:
I20260620 11:44:57.611828 20843 leader_election.cc:288] T 00000000000000000000000000000000 P 4195d863c1404c0a8137a88b6d8ad6f1 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260620 11:44:57.611865 20848 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 4195d863c1404c0a8137a88b6d8ad6f1 [term 1 FOLLOWER]: Leader election won for term 1
I20260620 11:44:57.611980 20848 raft_consensus.cc:686] T 00000000000000000000000000000000 P 4195d863c1404c0a8137a88b6d8ad6f1 [term 1 LEADER]: Becoming Leader. State: Replica: 4195d863c1404c0a8137a88b6d8ad6f1, State: Running, Role: LEADER
I20260620 11:44:57.612093 20843 sys_catalog.cc:564] T 00000000000000000000000000000000 P 4195d863c1404c0a8137a88b6d8ad6f1 [sys.catalog]: configured and running, proceeding with master startup.
I20260620 11:44:57.612108 20848 consensus_queue.cc:237] T 00000000000000000000000000000000 P 4195d863c1404c0a8137a88b6d8ad6f1 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4195d863c1404c0a8137a88b6d8ad6f1" member_type: VOTER last_known_addr { host: "127.6.234.254" port: 44633 } }
I20260620 11:44:57.612564 20849 sys_catalog.cc:455] T 00000000000000000000000000000000 P 4195d863c1404c0a8137a88b6d8ad6f1 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "4195d863c1404c0a8137a88b6d8ad6f1" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4195d863c1404c0a8137a88b6d8ad6f1" member_type: VOTER last_known_addr { host: "127.6.234.254" port: 44633 } } }
I20260620 11:44:57.612663 20849 sys_catalog.cc:458] T 00000000000000000000000000000000 P 4195d863c1404c0a8137a88b6d8ad6f1 [sys.catalog]: This master's current role is: LEADER
I20260620 11:44:57.612886 20850 sys_catalog.cc:455] T 00000000000000000000000000000000 P 4195d863c1404c0a8137a88b6d8ad6f1 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 4195d863c1404c0a8137a88b6d8ad6f1. Latest consensus state: current_term: 1 leader_uuid: "4195d863c1404c0a8137a88b6d8ad6f1" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4195d863c1404c0a8137a88b6d8ad6f1" member_type: VOTER last_known_addr { host: "127.6.234.254" port: 44633 } } }
I20260620 11:44:57.613154 20850 sys_catalog.cc:458] T 00000000000000000000000000000000 P 4195d863c1404c0a8137a88b6d8ad6f1 [sys.catalog]: This master's current role is: LEADER
I20260620 11:44:57.613312 20855 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260620 11:44:57.613886 20855 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260620 11:44:57.615578 20855 catalog_manager.cc:1347] Generated new cluster ID: 19042a500bc94fdabc05bb1fbd473859
I20260620 11:44:57.615634 20855 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
Jun 20 11:44:57 dist-test-slave-cjc6 krb5kdc[20762](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781955897, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.6.234.254@KRBTEST.COM
I20260620 11:44:57.620052 20847 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 11:44:57.609157 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52292 (local address 127.6.234.254:44633)
0620 11:44:57.609350 (+ 193us) server_negotiation.cc:206] Beginning negotiation
0620 11:44:57.609359 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0620 11:44:57.609397 (+ 38us) server_negotiation.cc:407] Connection header received
0620 11:44:57.610306 (+ 909us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 11:44:57.610313 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 11:44:57.610416 (+ 103us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 11:44:57.610579 (+ 163us) server_negotiation.cc:226] Negotiated authn=SASL
0620 11:44:57.611330 (+ 751us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 11:44:57.612348 (+ 1018us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 11:44:57.613452 (+ 1104us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 11:44:57.614107 (+ 655us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 11:44:57.614223 (+ 116us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 11:44:57.616809 (+ 2586us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 11:44:57.616820 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 11:44:57.616821 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 11:44:57.616832 (+ 11us) server_negotiation.cc:922] calling sasl_server_start()
0620 11:44:57.618556 (+ 1724us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 11:44:57.618941 (+ 385us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 11:44:57.618944 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 11:44:57.618944 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 11:44:57.619033 (+ 89us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 11:44:57.619246 (+ 213us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 11:44:57.619247 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 11:44:57.619248 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 11:44:57.619478 (+ 230us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 11:44:57.619641 (+ 163us) server_negotiation.cc:1035] Waiting for connection context
0620 11:44:57.619761 (+ 120us) server_negotiation.cc:299] Negotiation successful
0620 11:44:57.619905 (+ 144us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":141,"thread_start_us":87,"threads_started":1}
I20260620 11:44:57.630897 20855 catalog_manager.cc:1370] Generated new certificate authority record
I20260620 11:44:57.631595 20855 catalog_manager.cc:1504] Loading token signing keys...
I20260620 11:44:57.634785 20855 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 4195d863c1404c0a8137a88b6d8ad6f1: Generated new TSK 0
I20260620 11:44:57.634953 20855 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.6.234.193@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.6.234.193@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.6.234.193 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task1JRg7N/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781955746769051-7083-0/krb5kdc/kudu_127.6.234.193.keytab.
Entry for principal kudu/127.6.234.193 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task1JRg7N/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781955746769051-7083-0/krb5kdc/kudu_127.6.234.193.keytab.
I20260620 11:44:57.665966 7083 external_mini_cluster.cc:1326] Running /tmp/dist-test-task1JRg7N/build/release/bin/kudu
/tmp/dist-test-task1JRg7N/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task1JRg7N/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781955746769051-7083-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-task1JRg7N/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781955746769051-7083-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task1JRg7N/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781955746769051-7083-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-task1JRg7N/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781955746769051-7083-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.6.234.193:0
--local_ip_for_outbound_sockets=127.6.234.193
--webserver_interface=127.6.234.193
--webserver_port=0
--tserver_master_addrs=127.6.234.254:44633
--keytab_file=/tmp/dist-test-task1JRg7N/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781955746769051-7083-0/krb5kdc/kudu_127.6.234.193.keytab
--principal=kudu/127.6.234.193
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.6.234.212:34463
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task1JRg7N/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781955746769051-7083-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task1JRg7N/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781955746769051-7083-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task1JRg7N/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781955746769051-7083-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 11:44:57.746690 20869 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 11:44:57.746935 20869 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 11:44:57.746973 20869 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 11:44:57.748114 20869 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 11:44:57.748185 20869 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 11:44:57.748232 20869 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.6.234.193
I20260620 11:44:57.749750 20869 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.6.234.212:34463
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task1JRg7N/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781955746769051-7083-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-task1JRg7N/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781955746769051-7083-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task1JRg7N/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781955746769051-7083-0/krb5kdc/kudu_127.6.234.193.keytab
--principal=kudu/127.6.234.193
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.6.234.193:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task1JRg7N/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781955746769051-7083-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.6.234.193
--webserver_port=0
--tserver_master_addrs=127.6.234.254:44633
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.20869
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.6.234.193
--log_dir=/tmp/dist-test-task1JRg7N/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781955746769051-7083-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 20 Jun 2026 10:32:37 UTC on ccycloud.quasar-bgyofv.root.comops.site
build id 4457173
I20260620 11:44:57.750025 20869 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 11:44:57.750262 20869 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 11:44:57.755113 20875 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 11:44:57.755131 20877 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 11:44:57.755122 20874 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 11:44:57.755539 20869 server_base.cc:1034] running on GCE node
I20260620 11:44:57.755745 20869 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 11:44:57.755985 20869 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 11:44:57.757120 20869 hybrid_clock.cc:648] HybridClock initialized: now 1781955897757110 us; error 26 us; skew 500 ppm
Jun 20 11:44:57 dist-test-slave-cjc6 krb5kdc[20762](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781955897, etypes {rep=17 tkt=17 ses=17}, kudu/127.6.234.193@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 11:44:57.759670 20869 init.cc:377] Logged in from keytab as kudu/127.6.234.193@KRBTEST.COM (short username kudu)
I20260620 11:44:57.760119 20869 webserver.cc:460] Webserver started at http://127.6.234.193:45589/ using document root <none> and password file <none>
I20260620 11:44:57.760301 20869 fs_manager.cc:359] Metadata directory not provided
I20260620 11:44:57.760344 20869 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 11:44:57.760429 20869 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 11:44:57.761374 20869 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task1JRg7N/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781955746769051-7083-0/minicluster-data/ts-0/data/instance:
uuid: "6705ef55d9584b59865e2da234933258"
format_stamp: "Formatted at 2026-06-20 11:44:57 on dist-test-slave-cjc6"
server_key: "aaecbf620d1e90b359cbec9b916a3ad9"
server_key_iv: "36d40e1318d339ab95f1f04a89b23817"
server_key_version: "encryptionkey@0"
I20260620 11:44:57.761669 20869 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task1JRg7N/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781955746769051-7083-0/minicluster-data/ts-0/wal/instance:
uuid: "6705ef55d9584b59865e2da234933258"
format_stamp: "Formatted at 2026-06-20 11:44:57 on dist-test-slave-cjc6"
server_key: "aaecbf620d1e90b359cbec9b916a3ad9"
server_key_iv: "36d40e1318d339ab95f1f04a89b23817"
server_key_version: "encryptionkey@0"
I20260620 11:44:57.762959 20869 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.001s sys 0.000s
I20260620 11:44:57.763625 20884 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 11:44:57.763797 20869 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260620 11:44:57.763901 20869 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task1JRg7N/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781955746769051-7083-0/minicluster-data/ts-0/data,/tmp/dist-test-task1JRg7N/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781955746769051-7083-0/minicluster-data/ts-0/wal
uuid: "6705ef55d9584b59865e2da234933258"
format_stamp: "Formatted at 2026-06-20 11:44:57 on dist-test-slave-cjc6"
server_key: "aaecbf620d1e90b359cbec9b916a3ad9"
server_key_iv: "36d40e1318d339ab95f1f04a89b23817"
server_key_version: "encryptionkey@0"
I20260620 11:44:57.763980 20869 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task1JRg7N/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781955746769051-7083-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-task1JRg7N/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781955746769051-7083-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-task1JRg7N/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781955746769051-7083-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 11:44:57.778064 20869 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 11:44:57.780426 20869 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 11:44:57.780592 20869 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 11:44:57.780835 20869 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 11:44:57.781147 20869 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 11:44:57.781200 20869 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 11:44:57.781245 20869 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 11:44:57.781270 20869 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 11:44:57.787662 20869 rpc_server.cc:307] RPC server started. Bound to: 127.6.234.193:35733
I20260620 11:44:57.787709 20997 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.6.234.193:35733 every 8 connection(s)
I20260620 11:44:57.788074 20869 server_base.cc:1166] Dumped server information to /tmp/dist-test-task1JRg7N/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781955746769051-7083-0/minicluster-data/ts-0/data/info.pb
I20260620 11:44:57.790769 7083 external_mini_cluster.cc:1388] Started /tmp/dist-test-task1JRg7N/build/release/bin/kudu as pid 20869
I20260620 11:44:57.790865 7083 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task1JRg7N/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781955746769051-7083-0/minicluster-data/ts-0/wal/instance
I20260620 11:44:57.791069 7083 external_mini_cluster.cc:1428] Setting key 80c695482734ba9973e1c6b1bb4010f3
Jun 20 11:44:57 dist-test-slave-cjc6 krb5kdc[20762](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781955897, etypes {rep=17 tkt=17 ses=17}, kudu/127.6.234.193@KRBTEST.COM for kudu/127.6.234.254@KRBTEST.COM
I20260620 11:44:57.798656 21000 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 11:44:57.788816 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.234.254:44633 (local address 127.6.234.193:40307)
0620 11:44:57.788975 (+ 159us) negotiation.cc:107] Waiting for socket to connect
0620 11:44:57.789022 (+ 47us) client_negotiation.cc:174] Beginning negotiation
0620 11:44:57.789492 (+ 470us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 11:44:57.789685 (+ 193us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 11:44:57.789688 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 11:44:57.790261 (+ 573us) client_negotiation.cc:189] Negotiated authn=SASL
0620 11:44:57.790455 (+ 194us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 11:44:57.790460 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 11:44:57.791327 (+ 867us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 11:44:57.791328 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 11:44:57.792594 (+ 1266us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 11:44:57.792597 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 11:44:57.792964 (+ 367us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 11:44:57.792965 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 11:44:57.793112 (+ 147us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 11:44:57.793748 (+ 636us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 11:44:57.793759 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0620 11:44:57.795339 (+ 1580us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 11:44:57.797286 (+ 1947us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 11:44:57.797293 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 11:44:57.797295 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0620 11:44:57.797503 (+ 208us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 11:44:57.797767 (+ 264us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 11:44:57.797768 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 11:44:57.797769 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0620 11:44:57.797871 (+ 102us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 11:44:57.798179 (+ 308us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 11:44:57.798183 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 11:44:57.798297 (+ 114us) client_negotiation.cc:769] Sending connection context
0620 11:44:57.798379 (+ 82us) client_negotiation.cc:240] Negotiation successful
0620 11:44:57.798495 (+ 116us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":117,"thread_start_us":89,"threads_started":1}
I20260620 11:44:57.798655 20847 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 11:44:57.788829 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.6.234.193:40307 (local address 127.6.234.254:44633)
0620 11:44:57.788891 (+ 62us) server_negotiation.cc:206] Beginning negotiation
0620 11:44:57.788893 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 11:44:57.789114 (+ 221us) server_negotiation.cc:407] Connection header received
0620 11:44:57.789567 (+ 453us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 11:44:57.789568 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 11:44:57.789577 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 11:44:57.789633 (+ 56us) server_negotiation.cc:226] Negotiated authn=SASL
0620 11:44:57.790536 (+ 903us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 11:44:57.791231 (+ 695us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 11:44:57.792676 (+ 1445us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 11:44:57.792882 (+ 206us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 11:44:57.792937 (+ 55us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 11:44:57.795435 (+ 2498us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 11:44:57.795437 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 11:44:57.795438 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 11:44:57.795446 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0620 11:44:57.797187 (+ 1741us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 11:44:57.797577 (+ 390us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 11:44:57.797578 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 11:44:57.797579 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 11:44:57.797662 (+ 83us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 11:44:57.797942 (+ 280us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 11:44:57.797943 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 11:44:57.797943 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 11:44:57.798102 (+ 159us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 11:44:57.798178 (+ 76us) server_negotiation.cc:1035] Waiting for connection context
0620 11:44:57.798431 (+ 253us) server_negotiation.cc:299] Negotiation successful
0620 11:44:57.798526 (+ 95us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":36}
I20260620 11:44:57.799234 20998 heartbeater.cc:347] Connected to a master server at 127.6.234.254:44633
I20260620 11:44:57.799351 20998 heartbeater.cc:464] Registering TS with master...
I20260620 11:44:57.799526 20998 heartbeater.cc:511] Master 127.6.234.254:44633 requested a full tablet report, sending...
I20260620 11:44:57.800091 20807 ts_manager.cc:194] Registered new tserver with Master: 6705ef55d9584b59865e2da234933258 (127.6.234.193:35733)
WARNING: no policy specified for kudu/127.6.234.194@KRBTEST.COM; defaulting to no policy
I20260620 11:44:57.800701 20807 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.6.234.193@KRBTEST.COM'} at 127.6.234.193:40307
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.6.234.194@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.6.234.194 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task1JRg7N/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781955746769051-7083-0/krb5kdc/kudu_127.6.234.194.keytab.
Entry for principal kudu/127.6.234.194 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task1JRg7N/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781955746769051-7083-0/krb5kdc/kudu_127.6.234.194.keytab.
I20260620 11:44:57.816421 7083 external_mini_cluster.cc:1326] Running /tmp/dist-test-task1JRg7N/build/release/bin/kudu
/tmp/dist-test-task1JRg7N/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task1JRg7N/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781955746769051-7083-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-task1JRg7N/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781955746769051-7083-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task1JRg7N/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781955746769051-7083-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-task1JRg7N/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781955746769051-7083-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.6.234.194:0
--local_ip_for_outbound_sockets=127.6.234.194
--webserver_interface=127.6.234.194
--webserver_port=0
--tserver_master_addrs=127.6.234.254:44633
--keytab_file=/tmp/dist-test-task1JRg7N/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781955746769051-7083-0/krb5kdc/kudu_127.6.234.194.keytab
--principal=kudu/127.6.234.194
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.6.234.212:34463
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task1JRg7N/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781955746769051-7083-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task1JRg7N/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781955746769051-7083-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task1JRg7N/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781955746769051-7083-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 11:44:57.896299 21003 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 11:44:57.896503 21003 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 11:44:57.896526 21003 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 11:44:57.897593 21003 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 11:44:57.897650 21003 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 11:44:57.897676 21003 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.6.234.194
I20260620 11:44:57.899151 21003 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.6.234.212:34463
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task1JRg7N/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781955746769051-7083-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-task1JRg7N/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781955746769051-7083-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task1JRg7N/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781955746769051-7083-0/krb5kdc/kudu_127.6.234.194.keytab
--principal=kudu/127.6.234.194
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.6.234.194:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task1JRg7N/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781955746769051-7083-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.6.234.194
--webserver_port=0
--tserver_master_addrs=127.6.234.254:44633
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.21003
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.6.234.194
--log_dir=/tmp/dist-test-task1JRg7N/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781955746769051-7083-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 20 Jun 2026 10:32:37 UTC on ccycloud.quasar-bgyofv.root.comops.site
build id 4457173
I20260620 11:44:57.899333 21003 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 11:44:57.899505 21003 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 11:44:57.904450 21008 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 11:44:57.904448 21009 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 11:44:57.904474 21003 server_base.cc:1034] running on GCE node
W20260620 11:44:57.904439 21011 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 11:44:57.904809 21003 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 11:44:57.905055 21003 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 11:44:57.906221 21003 hybrid_clock.cc:648] HybridClock initialized: now 1781955897906211 us; error 32 us; skew 500 ppm
Jun 20 11:44:57 dist-test-slave-cjc6 krb5kdc[20762](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781955897, etypes {rep=17 tkt=17 ses=17}, kudu/127.6.234.194@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 11:44:57.908823 21003 init.cc:377] Logged in from keytab as kudu/127.6.234.194@KRBTEST.COM (short username kudu)
I20260620 11:44:57.909304 21003 webserver.cc:460] Webserver started at http://127.6.234.194:40019/ using document root <none> and password file <none>
I20260620 11:44:57.909531 21003 fs_manager.cc:359] Metadata directory not provided
I20260620 11:44:57.909600 21003 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 11:44:57.909782 21003 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 11:44:57.910810 21003 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task1JRg7N/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781955746769051-7083-0/minicluster-data/ts-1/data/instance:
uuid: "fd553ed4672d4f84a1058e83d69d0e88"
format_stamp: "Formatted at 2026-06-20 11:44:57 on dist-test-slave-cjc6"
server_key: "175cb27c7fe999d5d8de3c5eb757ab3a"
server_key_iv: "7a8318dc18ca4f5755ecaf130d21ff5d"
server_key_version: "encryptionkey@0"
I20260620 11:44:57.911249 21003 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task1JRg7N/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781955746769051-7083-0/minicluster-data/ts-1/wal/instance:
uuid: "fd553ed4672d4f84a1058e83d69d0e88"
format_stamp: "Formatted at 2026-06-20 11:44:57 on dist-test-slave-cjc6"
server_key: "175cb27c7fe999d5d8de3c5eb757ab3a"
server_key_iv: "7a8318dc18ca4f5755ecaf130d21ff5d"
server_key_version: "encryptionkey@0"
I20260620 11:44:57.912585 21003 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260620 11:44:57.913396 21018 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 11:44:57.913610 21003 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260620 11:44:57.913735 21003 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task1JRg7N/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781955746769051-7083-0/minicluster-data/ts-1/data,/tmp/dist-test-task1JRg7N/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781955746769051-7083-0/minicluster-data/ts-1/wal
uuid: "fd553ed4672d4f84a1058e83d69d0e88"
format_stamp: "Formatted at 2026-06-20 11:44:57 on dist-test-slave-cjc6"
server_key: "175cb27c7fe999d5d8de3c5eb757ab3a"
server_key_iv: "7a8318dc18ca4f5755ecaf130d21ff5d"
server_key_version: "encryptionkey@0"
I20260620 11:44:57.913808 21003 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task1JRg7N/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781955746769051-7083-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-task1JRg7N/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781955746769051-7083-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-task1JRg7N/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781955746769051-7083-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 11:44:57.922508 21003 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 11:44:57.924710 21003 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 11:44:57.924817 21003 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 11:44:57.924988 21003 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 11:44:57.925230 21003 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 11:44:57.925261 21003 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 11:44:57.925282 21003 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 11:44:57.925294 21003 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 11:44:57.931084 21003 rpc_server.cc:307] RPC server started. Bound to: 127.6.234.194:40017
I20260620 11:44:57.931133 21131 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.6.234.194:40017 every 8 connection(s)
I20260620 11:44:57.931442 21003 server_base.cc:1166] Dumped server information to /tmp/dist-test-task1JRg7N/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781955746769051-7083-0/minicluster-data/ts-1/data/info.pb
Jun 20 11:44:57 dist-test-slave-cjc6 krb5kdc[20762](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781955897, etypes {rep=17 tkt=17 ses=17}, kudu/127.6.234.194@KRBTEST.COM for kudu/127.6.234.254@KRBTEST.COM
I20260620 11:44:57.940150 20847 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 11:44:57.932056 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.6.234.194:46091 (local address 127.6.234.254:44633)
0620 11:44:57.932147 (+ 91us) server_negotiation.cc:206] Beginning negotiation
0620 11:44:57.932149 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 11:44:57.932364 (+ 215us) server_negotiation.cc:407] Connection header received
0620 11:44:57.932917 (+ 553us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 11:44:57.932918 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 11:44:57.932927 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 11:44:57.932981 (+ 54us) server_negotiation.cc:226] Negotiated authn=SASL
0620 11:44:57.934000 (+ 1019us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 11:44:57.934499 (+ 499us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 11:44:57.935480 (+ 981us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 11:44:57.935666 (+ 186us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 11:44:57.935725 (+ 59us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 11:44:57.937830 (+ 2105us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 11:44:57.937832 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 11:44:57.937832 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 11:44:57.937839 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0620 11:44:57.938994 (+ 1155us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 11:44:57.939401 (+ 407us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 11:44:57.939402 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 11:44:57.939402 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 11:44:57.939449 (+ 47us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 11:44:57.939656 (+ 207us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 11:44:57.939656 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 11:44:57.939656 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 11:44:57.939752 (+ 96us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 11:44:57.939801 (+ 49us) server_negotiation.cc:1035] Waiting for connection context
0620 11:44:57.940029 (+ 228us) server_negotiation.cc:299] Negotiation successful
0620 11:44:57.940087 (+ 58us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":59}
I20260620 11:44:57.940311 21134 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 11:44:57.932023 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.234.254:44633 (local address 127.6.234.194:46091)
0620 11:44:57.932205 (+ 182us) negotiation.cc:107] Waiting for socket to connect
0620 11:44:57.932240 (+ 35us) client_negotiation.cc:174] Beginning negotiation
0620 11:44:57.932822 (+ 582us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 11:44:57.933055 (+ 233us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 11:44:57.933058 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 11:44:57.933615 (+ 557us) client_negotiation.cc:189] Negotiated authn=SASL
0620 11:44:57.933921 (+ 306us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 11:44:57.933926 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 11:44:57.934582 (+ 656us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 11:44:57.934583 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 11:44:57.935396 (+ 813us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 11:44:57.935397 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 11:44:57.935746 (+ 349us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 11:44:57.935747 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 11:44:57.935842 (+ 95us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 11:44:57.936283 (+ 441us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 11:44:57.936291 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0620 11:44:57.937685 (+ 1394us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 11:44:57.939088 (+ 1403us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 11:44:57.939094 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 11:44:57.939095 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0620 11:44:57.939332 (+ 237us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 11:44:57.939508 (+ 176us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 11:44:57.939509 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 11:44:57.939509 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0620 11:44:57.939603 (+ 94us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 11:44:57.939806 (+ 203us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 11:44:57.939810 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 11:44:57.939922 (+ 112us) client_negotiation.cc:769] Sending connection context
0620 11:44:57.940002 (+ 80us) client_negotiation.cc:240] Negotiation successful
0620 11:44:57.940113 (+ 111us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":147,"thread_start_us":105,"threads_started":1}
I20260620 11:44:57.940814 21132 heartbeater.cc:347] Connected to a master server at 127.6.234.254:44633
I20260620 11:44:57.940922 21132 heartbeater.cc:464] Registering TS with master...
I20260620 11:44:57.941017 7083 external_mini_cluster.cc:1388] Started /tmp/dist-test-task1JRg7N/build/release/bin/kudu as pid 21003
I20260620 11:44:57.941118 21132 heartbeater.cc:511] Master 127.6.234.254:44633 requested a full tablet report, sending...
I20260620 11:44:57.941139 7083 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task1JRg7N/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781955746769051-7083-0/minicluster-data/ts-1/wal/instance
I20260620 11:44:57.941366 7083 external_mini_cluster.cc:1428] Setting key 3d76985655c3b3fff2f416749d7d8110
I20260620 11:44:57.941557 20807 ts_manager.cc:194] Registered new tserver with Master: fd553ed4672d4f84a1058e83d69d0e88 (127.6.234.194:40017)
I20260620 11:44:57.942199 20807 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.6.234.194@KRBTEST.COM'} at 127.6.234.194:46091
WARNING: no policy specified for kudu/127.6.234.195@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.6.234.195@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.6.234.195 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task1JRg7N/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781955746769051-7083-0/krb5kdc/kudu_127.6.234.195.keytab.
Entry for principal kudu/127.6.234.195 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task1JRg7N/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781955746769051-7083-0/krb5kdc/kudu_127.6.234.195.keytab.
I20260620 11:44:57.964526 7083 external_mini_cluster.cc:1326] Running /tmp/dist-test-task1JRg7N/build/release/bin/kudu
/tmp/dist-test-task1JRg7N/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task1JRg7N/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781955746769051-7083-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-task1JRg7N/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781955746769051-7083-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task1JRg7N/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781955746769051-7083-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-task1JRg7N/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781955746769051-7083-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.6.234.195:0
--local_ip_for_outbound_sockets=127.6.234.195
--webserver_interface=127.6.234.195
--webserver_port=0
--tserver_master_addrs=127.6.234.254:44633
--keytab_file=/tmp/dist-test-task1JRg7N/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781955746769051-7083-0/krb5kdc/kudu_127.6.234.195.keytab
--principal=kudu/127.6.234.195
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.6.234.212:34463
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task1JRg7N/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781955746769051-7083-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task1JRg7N/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781955746769051-7083-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task1JRg7N/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781955746769051-7083-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 11:44:58.045771 21137 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 11:44:58.045967 21137 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 11:44:58.045990 21137 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 11:44:58.047067 21137 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 11:44:58.047101 21137 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 11:44:58.047127 21137 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.6.234.195
I20260620 11:44:58.048450 21137 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.6.234.212:34463
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task1JRg7N/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781955746769051-7083-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-task1JRg7N/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781955746769051-7083-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task1JRg7N/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781955746769051-7083-0/krb5kdc/kudu_127.6.234.195.keytab
--principal=kudu/127.6.234.195
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.6.234.195:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task1JRg7N/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781955746769051-7083-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.6.234.195
--webserver_port=0
--tserver_master_addrs=127.6.234.254:44633
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.21137
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.6.234.195
--log_dir=/tmp/dist-test-task1JRg7N/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781955746769051-7083-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 20 Jun 2026 10:32:37 UTC on ccycloud.quasar-bgyofv.root.comops.site
build id 4457173
I20260620 11:44:58.048677 21137 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 11:44:58.048864 21137 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 11:44:58.053752 21142 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 11:44:58.053740 21143 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 11:44:58.053870 21137 server_base.cc:1034] running on GCE node
W20260620 11:44:58.053752 21145 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 11:44:58.054198 21137 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 11:44:58.054451 21137 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 11:44:58.055609 21137 hybrid_clock.cc:648] HybridClock initialized: now 1781955898055595 us; error 27 us; skew 500 ppm
Jun 20 11:44:58 dist-test-slave-cjc6 krb5kdc[20762](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781955898, etypes {rep=17 tkt=17 ses=17}, kudu/127.6.234.195@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 11:44:58.058127 21137 init.cc:377] Logged in from keytab as kudu/127.6.234.195@KRBTEST.COM (short username kudu)
I20260620 11:44:58.058540 21137 webserver.cc:460] Webserver started at http://127.6.234.195:35283/ using document root <none> and password file <none>
I20260620 11:44:58.058758 21137 fs_manager.cc:359] Metadata directory not provided
I20260620 11:44:58.058821 21137 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 11:44:58.058969 21137 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 11:44:58.059959 21137 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task1JRg7N/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781955746769051-7083-0/minicluster-data/ts-2/data/instance:
uuid: "741432935b6d42c38ab97037b7ab9b6f"
format_stamp: "Formatted at 2026-06-20 11:44:58 on dist-test-slave-cjc6"
server_key: "0fdbd66a6280cdebd7c4853a2e880ec0"
server_key_iv: "3a8b79d819f0fbd10c123d784ae2fdaf"
server_key_version: "encryptionkey@0"
I20260620 11:44:58.060305 21137 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task1JRg7N/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781955746769051-7083-0/minicluster-data/ts-2/wal/instance:
uuid: "741432935b6d42c38ab97037b7ab9b6f"
format_stamp: "Formatted at 2026-06-20 11:44:58 on dist-test-slave-cjc6"
server_key: "0fdbd66a6280cdebd7c4853a2e880ec0"
server_key_iv: "3a8b79d819f0fbd10c123d784ae2fdaf"
server_key_version: "encryptionkey@0"
I20260620 11:44:58.061600 21137 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.001s
I20260620 11:44:58.062309 21152 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 11:44:58.062467 21137 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.000s
I20260620 11:44:58.062563 21137 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task1JRg7N/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781955746769051-7083-0/minicluster-data/ts-2/data,/tmp/dist-test-task1JRg7N/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781955746769051-7083-0/minicluster-data/ts-2/wal
uuid: "741432935b6d42c38ab97037b7ab9b6f"
format_stamp: "Formatted at 2026-06-20 11:44:58 on dist-test-slave-cjc6"
server_key: "0fdbd66a6280cdebd7c4853a2e880ec0"
server_key_iv: "3a8b79d819f0fbd10c123d784ae2fdaf"
server_key_version: "encryptionkey@0"
I20260620 11:44:58.062644 21137 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task1JRg7N/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781955746769051-7083-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-task1JRg7N/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781955746769051-7083-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-task1JRg7N/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781955746769051-7083-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 11:44:58.078094 21137 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 11:44:58.080331 21137 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 11:44:58.080502 21137 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 11:44:58.080722 21137 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 11:44:58.081014 21137 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 11:44:58.081066 21137 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 11:44:58.081101 21137 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 11:44:58.081128 21137 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 11:44:58.087023 21137 rpc_server.cc:307] RPC server started. Bound to: 127.6.234.195:44517
I20260620 11:44:58.087072 21265 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.6.234.195:44517 every 8 connection(s)
I20260620 11:44:58.087387 21137 server_base.cc:1166] Dumped server information to /tmp/dist-test-task1JRg7N/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781955746769051-7083-0/minicluster-data/ts-2/data/info.pb
I20260620 11:44:58.089202 7083 external_mini_cluster.cc:1388] Started /tmp/dist-test-task1JRg7N/build/release/bin/kudu as pid 21137
I20260620 11:44:58.089344 7083 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task1JRg7N/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781955746769051-7083-0/minicluster-data/ts-2/wal/instance
I20260620 11:44:58.089531 7083 external_mini_cluster.cc:1428] Setting key 25f1fc4048aae7c1fdeeaf1004a224ea
Jun 20 11:44:58 dist-test-slave-cjc6 krb5kdc[20762](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781955898, etypes {rep=17 tkt=17 ses=17}, kudu/127.6.234.195@KRBTEST.COM for kudu/127.6.234.254@KRBTEST.COM
I20260620 11:44:58.096541 20847 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 11:44:58.088091 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.6.234.195:35711 (local address 127.6.234.254:44633)
0620 11:44:58.088154 (+ 63us) server_negotiation.cc:206] Beginning negotiation
0620 11:44:58.088156 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 11:44:58.088373 (+ 217us) server_negotiation.cc:407] Connection header received
0620 11:44:58.088823 (+ 450us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 11:44:58.088824 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 11:44:58.088833 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 11:44:58.088891 (+ 58us) server_negotiation.cc:226] Negotiated authn=SASL
0620 11:44:58.089984 (+ 1093us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 11:44:58.090492 (+ 508us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 11:44:58.091484 (+ 992us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 11:44:58.091701 (+ 217us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 11:44:58.091779 (+ 78us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 11:44:58.093969 (+ 2190us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 11:44:58.093971 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 11:44:58.093971 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 11:44:58.093979 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0620 11:44:58.095159 (+ 1180us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 11:44:58.095562 (+ 403us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 11:44:58.095563 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 11:44:58.095564 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 11:44:58.095626 (+ 62us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 11:44:58.095886 (+ 260us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 11:44:58.095887 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 11:44:58.095887 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 11:44:58.096009 (+ 122us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 11:44:58.096084 (+ 75us) server_negotiation.cc:1035] Waiting for connection context
0620 11:44:58.096367 (+ 283us) server_negotiation.cc:299] Negotiation successful
0620 11:44:58.096447 (+ 80us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":37}
I20260620 11:44:58.096743 21268 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 11:44:58.088080 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.234.254:44633 (local address 127.6.234.195:35711)
0620 11:44:58.088238 (+ 158us) negotiation.cc:107] Waiting for socket to connect
0620 11:44:58.088278 (+ 40us) client_negotiation.cc:174] Beginning negotiation
0620 11:44:58.088741 (+ 463us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 11:44:58.088941 (+ 200us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 11:44:58.088944 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 11:44:58.089584 (+ 640us) client_negotiation.cc:189] Negotiated authn=SASL
0620 11:44:58.089889 (+ 305us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 11:44:58.089895 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 11:44:58.090588 (+ 693us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 11:44:58.090589 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 11:44:58.091389 (+ 800us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 11:44:58.091391 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 11:44:58.091783 (+ 392us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 11:44:58.091784 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 11:44:58.091916 (+ 132us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 11:44:58.092443 (+ 527us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 11:44:58.092451 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0620 11:44:58.093874 (+ 1423us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 11:44:58.095254 (+ 1380us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 11:44:58.095260 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 11:44:58.095262 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0620 11:44:58.095484 (+ 222us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 11:44:58.095704 (+ 220us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 11:44:58.095704 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 11:44:58.095705 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0620 11:44:58.095810 (+ 105us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 11:44:58.096086 (+ 276us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 11:44:58.096093 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 11:44:58.096228 (+ 135us) client_negotiation.cc:769] Sending connection context
0620 11:44:58.096329 (+ 101us) client_negotiation.cc:240] Negotiation successful
0620 11:44:58.096494 (+ 165us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":119,"thread_start_us":92,"threads_started":1}
I20260620 11:44:58.097187 21266 heartbeater.cc:347] Connected to a master server at 127.6.234.254:44633
I20260620 11:44:58.097262 21266 heartbeater.cc:464] Registering TS with master...
I20260620 11:44:58.097396 21266 heartbeater.cc:511] Master 127.6.234.254:44633 requested a full tablet report, sending...
I20260620 11:44:58.097788 20807 ts_manager.cc:194] Registered new tserver with Master: 741432935b6d42c38ab97037b7ab9b6f (127.6.234.195:44517)
I20260620 11:44:58.098196 20807 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.6.234.195@KRBTEST.COM'} at 127.6.234.195:35711
I20260620 11:44:58.101126 7083 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260620 11:44:58.108167 20847 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 11:44:58.102589 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52296 (local address 127.6.234.254:44633)
0620 11:44:58.102633 (+ 44us) server_negotiation.cc:206] Beginning negotiation
0620 11:44:58.102636 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0620 11:44:58.102793 (+ 157us) server_negotiation.cc:407] Connection header received
0620 11:44:58.102838 (+ 45us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 11:44:58.102839 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 11:44:58.102849 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 11:44:58.102895 (+ 46us) server_negotiation.cc:226] Negotiated authn=SASL
0620 11:44:58.103420 (+ 525us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 11:44:58.103877 (+ 457us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 11:44:58.104538 (+ 661us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 11:44:58.104707 (+ 169us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 11:44:58.104752 (+ 45us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 11:44:58.105883 (+ 1131us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 11:44:58.105884 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 11:44:58.105884 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 11:44:58.105890 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0620 11:44:58.107222 (+ 1332us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 11:44:58.107508 (+ 286us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 11:44:58.107509 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 11:44:58.107509 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 11:44:58.107589 (+ 80us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 11:44:58.107740 (+ 151us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 11:44:58.107740 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 11:44:58.107740 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 11:44:58.107871 (+ 131us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 11:44:58.107943 (+ 72us) server_negotiation.cc:1035] Waiting for connection context
0620 11:44:58.108025 (+ 82us) server_negotiation.cc:299] Negotiation successful
0620 11:44:58.108082 (+ 57us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":19}
I20260620 11:44:58.109573 20807 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:52296:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260620 11:44:58.111245 20807 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260620 11:44:58.117890 21276 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 11:44:58.113540 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57442 (local address 127.6.234.194:40017)
0620 11:44:58.114164 (+ 624us) server_negotiation.cc:206] Beginning negotiation
0620 11:44:58.114168 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 11:44:58.114177 (+ 9us) server_negotiation.cc:407] Connection header received
0620 11:44:58.114211 (+ 34us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 11:44:58.114213 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 11:44:58.114226 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 11:44:58.114389 (+ 163us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 11:44:58.114591 (+ 202us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 11:44:58.115583 (+ 992us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 11:44:58.116695 (+ 1112us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 11:44:58.117464 (+ 769us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 11:44:58.117572 (+ 108us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 11:44:58.117646 (+ 74us) server_negotiation.cc:1035] Waiting for connection context
0620 11:44:58.117766 (+ 120us) server_negotiation.cc:299] Negotiation successful
0620 11:44:58.117807 (+ 41us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":587,"thread_start_us":88,"threads_started":1}
I20260620 11:44:58.117887 21277 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 11:44:58.113551 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52738 (local address 127.6.234.193:35733)
0620 11:44:58.113803 (+ 252us) server_negotiation.cc:206] Beginning negotiation
0620 11:44:58.113809 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0620 11:44:58.114025 (+ 216us) server_negotiation.cc:407] Connection header received
0620 11:44:58.114123 (+ 98us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 11:44:58.114125 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 11:44:58.114141 (+ 16us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 11:44:58.114215 (+ 74us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 11:44:58.114489 (+ 274us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 11:44:58.115206 (+ 717us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 11:44:58.116729 (+ 1523us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 11:44:58.117298 (+ 569us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 11:44:58.117333 (+ 35us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 11:44:58.117378 (+ 45us) server_negotiation.cc:1035] Waiting for connection context
0620 11:44:58.117754 (+ 376us) server_negotiation.cc:299] Negotiation successful
0620 11:44:58.117801 (+ 47us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":217,"thread_start_us":91,"threads_started":1}
I20260620 11:44:58.118077 21279 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 11:44:58.113798 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48654 (local address 127.6.234.195:44517)
0620 11:44:58.113961 (+ 163us) server_negotiation.cc:206] Beginning negotiation
0620 11:44:58.113966 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 11:44:58.113978 (+ 12us) server_negotiation.cc:407] Connection header received
0620 11:44:58.114017 (+ 39us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 11:44:58.114020 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 11:44:58.114036 (+ 16us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 11:44:58.114120 (+ 84us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 11:44:58.114286 (+ 166us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 11:44:58.115367 (+ 1081us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 11:44:58.116955 (+ 1588us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 11:44:58.117608 (+ 653us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 11:44:58.117686 (+ 78us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 11:44:58.117758 (+ 72us) server_negotiation.cc:1035] Waiting for connection context
0620 11:44:58.117929 (+ 171us) server_negotiation.cc:299] Negotiation successful
0620 11:44:58.117976 (+ 47us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":133,"thread_start_us":99,"threads_started":1}
I20260620 11:44:58.118350 21275 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 11:44:58.113615 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.234.195:44517 (local address 127.0.0.1:48654)
0620 11:44:58.113846 (+ 231us) negotiation.cc:107] Waiting for socket to connect
0620 11:44:58.113855 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0620 11:44:58.113897 (+ 42us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 11:44:58.114132 (+ 235us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 11:44:58.114135 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 11:44:58.114138 (+ 3us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 11:44:58.114209 (+ 71us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 11:44:58.114211 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 11:44:58.115471 (+ 1260us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 11:44:58.115472 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 11:44:58.116880 (+ 1408us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 11:44:58.116882 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 11:44:58.117746 (+ 864us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 11:44:58.117747 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 11:44:58.117821 (+ 74us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 11:44:58.117825 (+ 4us) client_negotiation.cc:769] Sending connection context
0620 11:44:58.117912 (+ 87us) client_negotiation.cc:240] Negotiation successful
0620 11:44:58.118294 (+ 382us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":220,"mutex_wait_us":31,"thread_start_us":52,"threads_started":1}
I20260620 11:44:58.118479 21274 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 11:44:58.113414 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.234.194:40017 (local address 127.0.0.1:57442)
0620 11:44:58.113751 (+ 337us) negotiation.cc:107] Waiting for socket to connect
0620 11:44:58.113793 (+ 42us) client_negotiation.cc:174] Beginning negotiation
0620 11:44:58.113878 (+ 85us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 11:44:58.114466 (+ 588us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 11:44:58.114467 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 11:44:58.114470 (+ 3us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 11:44:58.114511 (+ 41us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 11:44:58.114512 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 11:44:58.115675 (+ 1163us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 11:44:58.115676 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 11:44:58.116613 (+ 937us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 11:44:58.116616 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 11:44:58.117570 (+ 954us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 11:44:58.117570 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 11:44:58.117640 (+ 70us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 11:44:58.117644 (+ 4us) client_negotiation.cc:769] Sending connection context
0620 11:44:58.117993 (+ 349us) client_negotiation.cc:240] Negotiation successful
0620 11:44:58.118337 (+ 344us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":308,"mutex_wait_us":268,"thread_start_us":53,"threads_started":1}
I20260620 11:44:58.118482 21278 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 11:44:58.113414 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.234.193:35733 (local address 127.0.0.1:52738)
0620 11:44:58.113947 (+ 533us) negotiation.cc:107] Waiting for socket to connect
0620 11:44:58.113959 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0620 11:44:58.114025 (+ 66us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 11:44:58.114296 (+ 271us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 11:44:58.114297 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 11:44:58.114299 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 11:44:58.114338 (+ 39us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 11:44:58.114339 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 11:44:58.115297 (+ 958us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 11:44:58.115298 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 11:44:58.116644 (+ 1346us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 11:44:58.116647 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 11:44:58.117464 (+ 817us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 11:44:58.117465 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 11:44:58.117563 (+ 98us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 11:44:58.117577 (+ 14us) client_negotiation.cc:769] Sending connection context
0620 11:44:58.118006 (+ 429us) client_negotiation.cc:240] Negotiation successful
0620 11:44:58.118338 (+ 332us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":333,"mutex_wait_us":326,"thread_start_us":46,"threads_started":1}
I20260620 11:44:58.119222 20932 tablet_service.cc:1467] Processing CreateTablet for tablet b5b6c99c41d54a56b52e4e7bbfe1cefa (DEFAULT_TABLE table=test-table [id=70697314689944c496e80b022368cd0d]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 11:44:58.119261 21200 tablet_service.cc:1467] Processing CreateTablet for tablet b5b6c99c41d54a56b52e4e7bbfe1cefa (DEFAULT_TABLE table=test-table [id=70697314689944c496e80b022368cd0d]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 11:44:58.119298 21066 tablet_service.cc:1467] Processing CreateTablet for tablet b5b6c99c41d54a56b52e4e7bbfe1cefa (DEFAULT_TABLE table=test-table [id=70697314689944c496e80b022368cd0d]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 11:44:58.119596 21200 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b5b6c99c41d54a56b52e4e7bbfe1cefa. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 11:44:58.119593 20932 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b5b6c99c41d54a56b52e4e7bbfe1cefa. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 11:44:58.119593 21066 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b5b6c99c41d54a56b52e4e7bbfe1cefa. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 11:44:58.122246 21280 tablet_bootstrap.cc:492] T b5b6c99c41d54a56b52e4e7bbfe1cefa P 741432935b6d42c38ab97037b7ab9b6f: Bootstrap starting.
I20260620 11:44:58.122866 21282 tablet_bootstrap.cc:492] T b5b6c99c41d54a56b52e4e7bbfe1cefa P fd553ed4672d4f84a1058e83d69d0e88: Bootstrap starting.
I20260620 11:44:58.123051 21280 tablet_bootstrap.cc:654] T b5b6c99c41d54a56b52e4e7bbfe1cefa P 741432935b6d42c38ab97037b7ab9b6f: Neither blocks nor log segments found. Creating new log.
I20260620 11:44:58.123447 21282 tablet_bootstrap.cc:654] T b5b6c99c41d54a56b52e4e7bbfe1cefa P fd553ed4672d4f84a1058e83d69d0e88: Neither blocks nor log segments found. Creating new log.
I20260620 11:44:58.123562 21280 log.cc:824] T b5b6c99c41d54a56b52e4e7bbfe1cefa P 741432935b6d42c38ab97037b7ab9b6f: Log is configured to *not* fsync() on all Append() calls
I20260620 11:44:58.124013 21282 log.cc:824] T b5b6c99c41d54a56b52e4e7bbfe1cefa P fd553ed4672d4f84a1058e83d69d0e88: Log is configured to *not* fsync() on all Append() calls
I20260620 11:44:58.124332 21281 tablet_bootstrap.cc:492] T b5b6c99c41d54a56b52e4e7bbfe1cefa P 6705ef55d9584b59865e2da234933258: Bootstrap starting.
I20260620 11:44:58.124409 21280 tablet_bootstrap.cc:492] T b5b6c99c41d54a56b52e4e7bbfe1cefa P 741432935b6d42c38ab97037b7ab9b6f: No bootstrap required, opened a new log
I20260620 11:44:58.124483 21280 ts_tablet_manager.cc:1397] T b5b6c99c41d54a56b52e4e7bbfe1cefa P 741432935b6d42c38ab97037b7ab9b6f: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20260620 11:44:58.124886 21281 tablet_bootstrap.cc:654] T b5b6c99c41d54a56b52e4e7bbfe1cefa P 6705ef55d9584b59865e2da234933258: Neither blocks nor log segments found. Creating new log.
I20260620 11:44:58.125416 21281 log.cc:824] T b5b6c99c41d54a56b52e4e7bbfe1cefa P 6705ef55d9584b59865e2da234933258: Log is configured to *not* fsync() on all Append() calls
I20260620 11:44:58.125999 21282 tablet_bootstrap.cc:492] T b5b6c99c41d54a56b52e4e7bbfe1cefa P fd553ed4672d4f84a1058e83d69d0e88: No bootstrap required, opened a new log
I20260620 11:44:58.126072 21282 ts_tablet_manager.cc:1397] T b5b6c99c41d54a56b52e4e7bbfe1cefa P fd553ed4672d4f84a1058e83d69d0e88: Time spent bootstrapping tablet: real 0.003s user 0.001s sys 0.000s
I20260620 11:44:58.126323 21281 tablet_bootstrap.cc:492] T b5b6c99c41d54a56b52e4e7bbfe1cefa P 6705ef55d9584b59865e2da234933258: No bootstrap required, opened a new log
I20260620 11:44:58.126322 21280 raft_consensus.cc:348] T b5b6c99c41d54a56b52e4e7bbfe1cefa P 741432935b6d42c38ab97037b7ab9b6f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fd553ed4672d4f84a1058e83d69d0e88" member_type: VOTER last_known_addr { host: "127.6.234.194" port: 40017 } } peers { permanent_uuid: "6705ef55d9584b59865e2da234933258" member_type: VOTER last_known_addr { host: "127.6.234.193" port: 35733 } } peers { permanent_uuid: "741432935b6d42c38ab97037b7ab9b6f" member_type: VOTER last_known_addr { host: "127.6.234.195" port: 44517 } }
I20260620 11:44:58.126431 21281 ts_tablet_manager.cc:1397] T b5b6c99c41d54a56b52e4e7bbfe1cefa P 6705ef55d9584b59865e2da234933258: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20260620 11:44:58.126489 21280 raft_consensus.cc:374] T b5b6c99c41d54a56b52e4e7bbfe1cefa P 741432935b6d42c38ab97037b7ab9b6f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 11:44:58.126534 21280 raft_consensus.cc:729] T b5b6c99c41d54a56b52e4e7bbfe1cefa P 741432935b6d42c38ab97037b7ab9b6f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 741432935b6d42c38ab97037b7ab9b6f, State: Initialized, Role: FOLLOWER
I20260620 11:44:58.126664 21280 consensus_queue.cc:260] T b5b6c99c41d54a56b52e4e7bbfe1cefa P 741432935b6d42c38ab97037b7ab9b6f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fd553ed4672d4f84a1058e83d69d0e88" member_type: VOTER last_known_addr { host: "127.6.234.194" port: 40017 } } peers { permanent_uuid: "6705ef55d9584b59865e2da234933258" member_type: VOTER last_known_addr { host: "127.6.234.193" port: 35733 } } peers { permanent_uuid: "741432935b6d42c38ab97037b7ab9b6f" member_type: VOTER last_known_addr { host: "127.6.234.195" port: 44517 } }
I20260620 11:44:58.126971 21280 ts_tablet_manager.cc:1428] T b5b6c99c41d54a56b52e4e7bbfe1cefa P 741432935b6d42c38ab97037b7ab9b6f: Time spent starting tablet: real 0.002s user 0.003s sys 0.000s
I20260620 11:44:58.126977 21266 heartbeater.cc:503] Master 127.6.234.254:44633 was elected leader, sending a full tablet report...
I20260620 11:44:58.128049 21282 raft_consensus.cc:348] T b5b6c99c41d54a56b52e4e7bbfe1cefa P fd553ed4672d4f84a1058e83d69d0e88 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fd553ed4672d4f84a1058e83d69d0e88" member_type: VOTER last_known_addr { host: "127.6.234.194" port: 40017 } } peers { permanent_uuid: "6705ef55d9584b59865e2da234933258" member_type: VOTER last_known_addr { host: "127.6.234.193" port: 35733 } } peers { permanent_uuid: "741432935b6d42c38ab97037b7ab9b6f" member_type: VOTER last_known_addr { host: "127.6.234.195" port: 44517 } }
I20260620 11:44:58.128242 21282 raft_consensus.cc:374] T b5b6c99c41d54a56b52e4e7bbfe1cefa P fd553ed4672d4f84a1058e83d69d0e88 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 11:44:58.128278 21282 raft_consensus.cc:729] T b5b6c99c41d54a56b52e4e7bbfe1cefa P fd553ed4672d4f84a1058e83d69d0e88 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: fd553ed4672d4f84a1058e83d69d0e88, State: Initialized, Role: FOLLOWER
I20260620 11:44:58.128403 21282 consensus_queue.cc:260] T b5b6c99c41d54a56b52e4e7bbfe1cefa P fd553ed4672d4f84a1058e83d69d0e88 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fd553ed4672d4f84a1058e83d69d0e88" member_type: VOTER last_known_addr { host: "127.6.234.194" port: 40017 } } peers { permanent_uuid: "6705ef55d9584b59865e2da234933258" member_type: VOTER last_known_addr { host: "127.6.234.193" port: 35733 } } peers { permanent_uuid: "741432935b6d42c38ab97037b7ab9b6f" member_type: VOTER last_known_addr { host: "127.6.234.195" port: 44517 } }
I20260620 11:44:58.128377 21281 raft_consensus.cc:348] T b5b6c99c41d54a56b52e4e7bbfe1cefa P 6705ef55d9584b59865e2da234933258 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fd553ed4672d4f84a1058e83d69d0e88" member_type: VOTER last_known_addr { host: "127.6.234.194" port: 40017 } } peers { permanent_uuid: "6705ef55d9584b59865e2da234933258" member_type: VOTER last_known_addr { host: "127.6.234.193" port: 35733 } } peers { permanent_uuid: "741432935b6d42c38ab97037b7ab9b6f" member_type: VOTER last_known_addr { host: "127.6.234.195" port: 44517 } }
I20260620 11:44:58.128547 21281 raft_consensus.cc:374] T b5b6c99c41d54a56b52e4e7bbfe1cefa P 6705ef55d9584b59865e2da234933258 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 11:44:58.128597 21281 raft_consensus.cc:729] T b5b6c99c41d54a56b52e4e7bbfe1cefa P 6705ef55d9584b59865e2da234933258 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6705ef55d9584b59865e2da234933258, State: Initialized, Role: FOLLOWER
I20260620 11:44:58.128682 21282 ts_tablet_manager.cc:1428] T b5b6c99c41d54a56b52e4e7bbfe1cefa P fd553ed4672d4f84a1058e83d69d0e88: Time spent starting tablet: real 0.003s user 0.000s sys 0.003s
I20260620 11:44:58.128695 21132 heartbeater.cc:503] Master 127.6.234.254:44633 was elected leader, sending a full tablet report...
I20260620 11:44:58.128731 21281 consensus_queue.cc:260] T b5b6c99c41d54a56b52e4e7bbfe1cefa P 6705ef55d9584b59865e2da234933258 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fd553ed4672d4f84a1058e83d69d0e88" member_type: VOTER last_known_addr { host: "127.6.234.194" port: 40017 } } peers { permanent_uuid: "6705ef55d9584b59865e2da234933258" member_type: VOTER last_known_addr { host: "127.6.234.193" port: 35733 } } peers { permanent_uuid: "741432935b6d42c38ab97037b7ab9b6f" member_type: VOTER last_known_addr { host: "127.6.234.195" port: 44517 } }
I20260620 11:44:58.129005 21281 ts_tablet_manager.cc:1428] T b5b6c99c41d54a56b52e4e7bbfe1cefa P 6705ef55d9584b59865e2da234933258: Time spent starting tablet: real 0.003s user 0.002s sys 0.000s
I20260620 11:44:58.129014 20998 heartbeater.cc:503] Master 127.6.234.254:44633 was elected leader, sending a full tablet report...
W20260620 11:44:58.181993 21133 tablet.cc:2369] T b5b6c99c41d54a56b52e4e7bbfe1cefa P fd553ed4672d4f84a1058e83d69d0e88: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260620 11:44:58.288827 20999 tablet.cc:2369] T b5b6c99c41d54a56b52e4e7bbfe1cefa P 6705ef55d9584b59865e2da234933258: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260620 11:44:58.337884 21267 tablet.cc:2369] T b5b6c99c41d54a56b52e4e7bbfe1cefa P 741432935b6d42c38ab97037b7ab9b6f: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260620 11:44:58.339161 21287 raft_consensus.cc:482] T b5b6c99c41d54a56b52e4e7bbfe1cefa P fd553ed4672d4f84a1058e83d69d0e88 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260620 11:44:58.339278 21287 raft_consensus.cc:504] T b5b6c99c41d54a56b52e4e7bbfe1cefa P fd553ed4672d4f84a1058e83d69d0e88 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fd553ed4672d4f84a1058e83d69d0e88" member_type: VOTER last_known_addr { host: "127.6.234.194" port: 40017 } } peers { permanent_uuid: "6705ef55d9584b59865e2da234933258" member_type: VOTER last_known_addr { host: "127.6.234.193" port: 35733 } } peers { permanent_uuid: "741432935b6d42c38ab97037b7ab9b6f" member_type: VOTER last_known_addr { host: "127.6.234.195" port: 44517 } }
I20260620 11:44:58.339682 21287 leader_election.cc:288] T b5b6c99c41d54a56b52e4e7bbfe1cefa P fd553ed4672d4f84a1058e83d69d0e88 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 6705ef55d9584b59865e2da234933258 (127.6.234.193:35733), 741432935b6d42c38ab97037b7ab9b6f (127.6.234.195:44517)
I20260620 11:44:58.343482 21289 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 11:44:58.339908 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.234.193:35733 (local address 127.6.234.194:56523)
0620 11:44:58.340088 (+ 180us) negotiation.cc:107] Waiting for socket to connect
0620 11:44:58.340101 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0620 11:44:58.340169 (+ 68us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 11:44:58.340386 (+ 217us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 11:44:58.340386 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 11:44:58.340388 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 11:44:58.340442 (+ 54us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 11:44:58.340442 (+ 0us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 11:44:58.341271 (+ 829us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 11:44:58.341271 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 11:44:58.342489 (+ 1218us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 11:44:58.342492 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 11:44:58.343218 (+ 726us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 11:44:58.343220 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 11:44:58.343297 (+ 77us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 11:44:58.343301 (+ 4us) client_negotiation.cc:769] Sending connection context
0620 11:44:58.343362 (+ 61us) client_negotiation.cc:240] Negotiation successful
0620 11:44:58.343395 (+ 33us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":160,"mutex_wait_us":39,"thread_start_us":109,"threads_started":1}
I20260620 11:44:58.343493 21277 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 11:44:58.340009 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.6.234.194:56523 (local address 127.6.234.193:35733)
0620 11:44:58.340111 (+ 102us) server_negotiation.cc:206] Beginning negotiation
0620 11:44:58.340114 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0620 11:44:58.340156 (+ 42us) server_negotiation.cc:407] Connection header received
0620 11:44:58.340279 (+ 123us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 11:44:58.340298 (+ 19us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 11:44:58.340309 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 11:44:58.340366 (+ 57us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 11:44:58.340509 (+ 143us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 11:44:58.341201 (+ 692us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 11:44:58.342583 (+ 1382us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 11:44:58.343129 (+ 546us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 11:44:58.343195 (+ 66us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 11:44:58.343203 (+ 8us) server_negotiation.cc:1035] Waiting for connection context
0620 11:44:58.343373 (+ 170us) server_negotiation.cc:299] Negotiation successful
0620 11:44:58.343410 (+ 37us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":76}
I20260620 11:44:58.343626 21279 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 11:44:58.339980 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.6.234.194:52273 (local address 127.6.234.195:44517)
0620 11:44:58.340044 (+ 64us) server_negotiation.cc:206] Beginning negotiation
0620 11:44:58.340046 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 11:44:58.340056 (+ 10us) server_negotiation.cc:407] Connection header received
0620 11:44:58.340086 (+ 30us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 11:44:58.340087 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 11:44:58.340097 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 11:44:58.340157 (+ 60us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 11:44:58.340330 (+ 173us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 11:44:58.341104 (+ 774us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 11:44:58.342609 (+ 1505us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 11:44:58.343220 (+ 611us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 11:44:58.343292 (+ 72us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 11:44:58.343301 (+ 9us) server_negotiation.cc:1035] Waiting for connection context
0620 11:44:58.343494 (+ 193us) server_negotiation.cc:299] Negotiation successful
0620 11:44:58.343540 (+ 46us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":43}
I20260620 11:44:58.343585 21134 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 11:44:58.339890 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.234.195:44517 (local address 127.6.234.194:52273)
0620 11:44:58.339969 (+ 79us) negotiation.cc:107] Waiting for socket to connect
0620 11:44:58.339981 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0620 11:44:58.340026 (+ 45us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 11:44:58.340169 (+ 143us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 11:44:58.340170 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 11:44:58.340172 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 11:44:58.340247 (+ 75us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 11:44:58.340249 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 11:44:58.341180 (+ 931us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 11:44:58.341181 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 11:44:58.342489 (+ 1308us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 11:44:58.342492 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 11:44:58.343299 (+ 807us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 11:44:58.343300 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 11:44:58.343408 (+ 108us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 11:44:58.343411 (+ 3us) client_negotiation.cc:769] Sending connection context
0620 11:44:58.343470 (+ 59us) client_negotiation.cc:240] Negotiation successful
0620 11:44:58.343508 (+ 38us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":58,"mutex_wait_us":41}
I20260620 11:44:58.344026 20952 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b5b6c99c41d54a56b52e4e7bbfe1cefa" candidate_uuid: "fd553ed4672d4f84a1058e83d69d0e88" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6705ef55d9584b59865e2da234933258" is_pre_election: true
I20260620 11:44:58.344070 21220 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b5b6c99c41d54a56b52e4e7bbfe1cefa" candidate_uuid: "fd553ed4672d4f84a1058e83d69d0e88" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "741432935b6d42c38ab97037b7ab9b6f" is_pre_election: true
I20260620 11:44:58.344188 20952 raft_consensus.cc:2454] T b5b6c99c41d54a56b52e4e7bbfe1cefa P 6705ef55d9584b59865e2da234933258 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate fd553ed4672d4f84a1058e83d69d0e88 in term 0.
I20260620 11:44:58.344187 21220 raft_consensus.cc:2454] T b5b6c99c41d54a56b52e4e7bbfe1cefa P 741432935b6d42c38ab97037b7ab9b6f [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate fd553ed4672d4f84a1058e83d69d0e88 in term 0.
I20260620 11:44:58.344424 21022 leader_election.cc:302] T b5b6c99c41d54a56b52e4e7bbfe1cefa P fd553ed4672d4f84a1058e83d69d0e88 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 6705ef55d9584b59865e2da234933258, fd553ed4672d4f84a1058e83d69d0e88; no voters:
I20260620 11:44:58.344574 21287 raft_consensus.cc:2781] T b5b6c99c41d54a56b52e4e7bbfe1cefa P fd553ed4672d4f84a1058e83d69d0e88 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260620 11:44:58.344620 21287 raft_consensus.cc:482] T b5b6c99c41d54a56b52e4e7bbfe1cefa P fd553ed4672d4f84a1058e83d69d0e88 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260620 11:44:58.344635 21287 raft_consensus.cc:3037] T b5b6c99c41d54a56b52e4e7bbfe1cefa P fd553ed4672d4f84a1058e83d69d0e88 [term 0 FOLLOWER]: Advancing to term 1
I20260620 11:44:58.345335 21287 raft_consensus.cc:504] T b5b6c99c41d54a56b52e4e7bbfe1cefa P fd553ed4672d4f84a1058e83d69d0e88 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fd553ed4672d4f84a1058e83d69d0e88" member_type: VOTER last_known_addr { host: "127.6.234.194" port: 40017 } } peers { permanent_uuid: "6705ef55d9584b59865e2da234933258" member_type: VOTER last_known_addr { host: "127.6.234.193" port: 35733 } } peers { permanent_uuid: "741432935b6d42c38ab97037b7ab9b6f" member_type: VOTER last_known_addr { host: "127.6.234.195" port: 44517 } }
I20260620 11:44:58.345507 21287 leader_election.cc:288] T b5b6c99c41d54a56b52e4e7bbfe1cefa P fd553ed4672d4f84a1058e83d69d0e88 [CANDIDATE]: Term 1 election: Requested vote from peers 6705ef55d9584b59865e2da234933258 (127.6.234.193:35733), 741432935b6d42c38ab97037b7ab9b6f (127.6.234.195:44517)
I20260620 11:44:58.345651 20952 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b5b6c99c41d54a56b52e4e7bbfe1cefa" candidate_uuid: "fd553ed4672d4f84a1058e83d69d0e88" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6705ef55d9584b59865e2da234933258"
I20260620 11:44:58.345669 21220 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b5b6c99c41d54a56b52e4e7bbfe1cefa" candidate_uuid: "fd553ed4672d4f84a1058e83d69d0e88" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "741432935b6d42c38ab97037b7ab9b6f"
I20260620 11:44:58.345757 21220 raft_consensus.cc:3037] T b5b6c99c41d54a56b52e4e7bbfe1cefa P 741432935b6d42c38ab97037b7ab9b6f [term 0 FOLLOWER]: Advancing to term 1
I20260620 11:44:58.345757 20952 raft_consensus.cc:3037] T b5b6c99c41d54a56b52e4e7bbfe1cefa P 6705ef55d9584b59865e2da234933258 [term 0 FOLLOWER]: Advancing to term 1
I20260620 11:44:58.346674 21220 raft_consensus.cc:2454] T b5b6c99c41d54a56b52e4e7bbfe1cefa P 741432935b6d42c38ab97037b7ab9b6f [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate fd553ed4672d4f84a1058e83d69d0e88 in term 1.
I20260620 11:44:58.346674 20952 raft_consensus.cc:2454] T b5b6c99c41d54a56b52e4e7bbfe1cefa P 6705ef55d9584b59865e2da234933258 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate fd553ed4672d4f84a1058e83d69d0e88 in term 1.
I20260620 11:44:58.346865 21022 leader_election.cc:302] T b5b6c99c41d54a56b52e4e7bbfe1cefa P fd553ed4672d4f84a1058e83d69d0e88 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 6705ef55d9584b59865e2da234933258, fd553ed4672d4f84a1058e83d69d0e88; no voters:
I20260620 11:44:58.346971 21287 raft_consensus.cc:2781] T b5b6c99c41d54a56b52e4e7bbfe1cefa P fd553ed4672d4f84a1058e83d69d0e88 [term 1 FOLLOWER]: Leader election won for term 1
I20260620 11:44:58.347143 21287 raft_consensus.cc:686] T b5b6c99c41d54a56b52e4e7bbfe1cefa P fd553ed4672d4f84a1058e83d69d0e88 [term 1 LEADER]: Becoming Leader. State: Replica: fd553ed4672d4f84a1058e83d69d0e88, State: Running, Role: LEADER
I20260620 11:44:58.347277 21287 consensus_queue.cc:237] T b5b6c99c41d54a56b52e4e7bbfe1cefa P fd553ed4672d4f84a1058e83d69d0e88 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fd553ed4672d4f84a1058e83d69d0e88" member_type: VOTER last_known_addr { host: "127.6.234.194" port: 40017 } } peers { permanent_uuid: "6705ef55d9584b59865e2da234933258" member_type: VOTER last_known_addr { host: "127.6.234.193" port: 35733 } } peers { permanent_uuid: "741432935b6d42c38ab97037b7ab9b6f" member_type: VOTER last_known_addr { host: "127.6.234.195" port: 44517 } }
I20260620 11:44:58.348044 20807 catalog_manager.cc:5495] T b5b6c99c41d54a56b52e4e7bbfe1cefa P fd553ed4672d4f84a1058e83d69d0e88 reported cstate change: term changed from 0 to 1, leader changed from <none> to fd553ed4672d4f84a1058e83d69d0e88 (127.6.234.194). New cstate: current_term: 1 leader_uuid: "fd553ed4672d4f84a1058e83d69d0e88" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fd553ed4672d4f84a1058e83d69d0e88" member_type: VOTER last_known_addr { host: "127.6.234.194" port: 40017 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "6705ef55d9584b59865e2da234933258" member_type: VOTER last_known_addr { host: "127.6.234.193" port: 35733 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "741432935b6d42c38ab97037b7ab9b6f" member_type: VOTER last_known_addr { host: "127.6.234.195" port: 44517 } health_report { overall_health: UNKNOWN } } }
I20260620 11:44:58.403568 21276 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 11:44:58.401025 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57454 (local address 127.6.234.194:40017)
0620 11:44:58.401092 (+ 67us) server_negotiation.cc:206] Beginning negotiation
0620 11:44:58.401094 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 11:44:58.401109 (+ 15us) server_negotiation.cc:407] Connection header received
0620 11:44:58.401144 (+ 35us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 11:44:58.401145 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 11:44:58.401163 (+ 18us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 11:44:58.401226 (+ 63us) server_negotiation.cc:226] Negotiated authn=TOKEN
0620 11:44:58.401367 (+ 141us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 11:44:58.401903 (+ 536us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 11:44:58.402696 (+ 793us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 11:44:58.402921 (+ 225us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 11:44:58.403001 (+ 80us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 11:44:58.403192 (+ 191us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0620 11:44:58.403313 (+ 121us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0620 11:44:58.403417 (+ 104us) server_negotiation.cc:1035] Waiting for connection context
0620 11:44:58.403436 (+ 19us) server_negotiation.cc:299] Negotiation successful
0620 11:44:58.403456 (+ 20us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":42}
I20260620 11:44:58.405654 21220 raft_consensus.cc:1261] T b5b6c99c41d54a56b52e4e7bbfe1cefa P 741432935b6d42c38ab97037b7ab9b6f [term 1 FOLLOWER]: Refusing update from remote peer fd553ed4672d4f84a1058e83d69d0e88: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 11:44:58.405654 20952 raft_consensus.cc:1261] T b5b6c99c41d54a56b52e4e7bbfe1cefa P 6705ef55d9584b59865e2da234933258 [term 1 FOLLOWER]: Refusing update from remote peer fd553ed4672d4f84a1058e83d69d0e88: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 11:44:58.406071 21287 consensus_queue.cc:1035] T b5b6c99c41d54a56b52e4e7bbfe1cefa P fd553ed4672d4f84a1058e83d69d0e88 [LEADER]: Connected to new peer: Peer: permanent_uuid: "741432935b6d42c38ab97037b7ab9b6f" member_type: VOTER last_known_addr { host: "127.6.234.195" port: 44517 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 11:44:58.406236 21290 consensus_queue.cc:1035] T b5b6c99c41d54a56b52e4e7bbfe1cefa P fd553ed4672d4f84a1058e83d69d0e88 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6705ef55d9584b59865e2da234933258" member_type: VOTER last_known_addr { host: "127.6.234.193" port: 35733 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 11:44:58.408412 21293 mvcc.cc:205] Tried to move back new op lower bound from 7298891359866540032 to 7298891359631020032. Current Snapshot: MvccSnapshot[applied={T|T < 7298891359866540032}]
I20260620 11:44:58.411020 20806 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:52296:
table { table_name: "test-table" } modify_external_catalogs: true
I20260620 11:44:58.411123 20806 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:52296:
table { table_name: "test-table" } modify_external_catalogs: true
I20260620 11:44:58.411877 20806 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 4195d863c1404c0a8137a88b6d8ad6f1: Sending DeleteTablet for 3 replicas of tablet b5b6c99c41d54a56b52e4e7bbfe1cefa
I20260620 11:44:58.412220 21066 tablet_service.cc:1514] Processing DeleteTablet for tablet b5b6c99c41d54a56b52e4e7bbfe1cefa with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 11:44:58 UTC) from {username='kudu', principal='kudu/127.6.234.254@KRBTEST.COM'} at 127.0.0.1:57442
I20260620 11:44:58.412248 20932 tablet_service.cc:1514] Processing DeleteTablet for tablet b5b6c99c41d54a56b52e4e7bbfe1cefa with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 11:44:58 UTC) from {username='kudu', principal='kudu/127.6.234.254@KRBTEST.COM'} at 127.0.0.1:52738
I20260620 11:44:58.412511 21303 tablet_replica.cc:330] T b5b6c99c41d54a56b52e4e7bbfe1cefa P 6705ef55d9584b59865e2da234933258: stopping tablet replica
I20260620 11:44:58.412597 21303 raft_consensus.cc:2229] T b5b6c99c41d54a56b52e4e7bbfe1cefa P 6705ef55d9584b59865e2da234933258 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260620 11:44:58.412562 21200 tablet_service.cc:1514] Processing DeleteTablet for tablet b5b6c99c41d54a56b52e4e7bbfe1cefa with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 11:44:58 UTC) from {username='kudu', principal='kudu/127.6.234.254@KRBTEST.COM'} at 127.0.0.1:48654
I20260620 11:44:58.412636 7083 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task1JRg7N/build/release/bin/kudu with pid 20869
I20260620 11:44:58.412676 21303 raft_consensus.cc:2258] T b5b6c99c41d54a56b52e4e7bbfe1cefa P 6705ef55d9584b59865e2da234933258 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 11:44:58.414206 21302 tablet_replica.cc:330] T b5b6c99c41d54a56b52e4e7bbfe1cefa P fd553ed4672d4f84a1058e83d69d0e88: stopping tablet replica
I20260620 11:44:58.414355 21302 raft_consensus.cc:2229] T b5b6c99c41d54a56b52e4e7bbfe1cefa P fd553ed4672d4f84a1058e83d69d0e88 [term 1 LEADER]: Raft consensus shutting down.
I20260620 11:44:58.414522 21302 raft_consensus.cc:2258] T b5b6c99c41d54a56b52e4e7bbfe1cefa P fd553ed4672d4f84a1058e83d69d0e88 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 11:44:58.415063 21302 ts_tablet_manager.cc:1905] T b5b6c99c41d54a56b52e4e7bbfe1cefa P fd553ed4672d4f84a1058e83d69d0e88: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 11:44:58.416823 21302 ts_tablet_manager.cc:1918] T b5b6c99c41d54a56b52e4e7bbfe1cefa P fd553ed4672d4f84a1058e83d69d0e88: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260620 11:44:58.416956 21302 log.cc:1196] T b5b6c99c41d54a56b52e4e7bbfe1cefa P fd553ed4672d4f84a1058e83d69d0e88: Deleting WAL directory at /tmp/dist-test-task1JRg7N/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781955746769051-7083-0/minicluster-data/ts-1/wal/wals/b5b6c99c41d54a56b52e4e7bbfe1cefa
I20260620 11:44:58.417308 21302 ts_tablet_manager.cc:1939] T b5b6c99c41d54a56b52e4e7bbfe1cefa P fd553ed4672d4f84a1058e83d69d0e88: Deleting consensus metadata
I20260620 11:44:58.417795 20792 catalog_manager.cc:4841] TS fd553ed4672d4f84a1058e83d69d0e88 (127.6.234.194:40017): tablet b5b6c99c41d54a56b52e4e7bbfe1cefa (table test-table [id=70697314689944c496e80b022368cd0d]) successfully deleted
W20260620 11:44:58.418035 20794 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.6.234.193:35733 (error 108)
I20260620 11:44:58.418397 7083 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task1JRg7N/build/release/bin/kudu with pid 21003
I20260620 11:44:58.418524 21275 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 11:44:58.418364 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.234.193:35733 (local address 127.0.0.1:52742)
0620 11:44:58.418427 (+ 63us) negotiation.cc:107] Waiting for socket to connect
0620 11:44:58.418484 (+ 57us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.234.193:35733: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":43}
W20260620 11:44:58.419355 20794 catalog_manager.cc:4584] TS 6705ef55d9584b59865e2da234933258 (127.6.234.193:35733): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet b5b6c99c41d54a56b52e4e7bbfe1cefa: Network error: Client connection negotiation failed: client connection to 127.6.234.193:35733: connect: Connection refused (error 111)
I20260620 11:44:58.423163 21304 tablet_replica.cc:330] T b5b6c99c41d54a56b52e4e7bbfe1cefa P 741432935b6d42c38ab97037b7ab9b6f: stopping tablet replica
I20260620 11:44:58.423290 21304 raft_consensus.cc:2229] T b5b6c99c41d54a56b52e4e7bbfe1cefa P 741432935b6d42c38ab97037b7ab9b6f [term 1 FOLLOWER]: Raft consensus shutting down.
I20260620 11:44:58.423413 21304 raft_consensus.cc:2258] T b5b6c99c41d54a56b52e4e7bbfe1cefa P 741432935b6d42c38ab97037b7ab9b6f [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 11:44:58.423430 7083 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task1JRg7N/build/release/bin/kudu with pid 21137
I20260620 11:44:58.428081 21275 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 11:44:58.427962 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.234.195:44517 (local address 127.0.0.1:48660)
0620 11:44:58.428018 (+ 56us) negotiation.cc:107] Waiting for socket to connect
0620 11:44:58.428045 (+ 27us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.234.195:44517: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":34}
I20260620 11:44:58.428078 7083 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task1JRg7N/build/release/bin/kudu with pid 20775
I20260620 11:44:58.433307 7083 external_mini_cluster.cc:1326] Running /tmp/dist-test-task1JRg7N/build/release/bin/kudu
/tmp/dist-test-task1JRg7N/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task1JRg7N/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781955746769051-7083-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task1JRg7N/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781955746769051-7083-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task1JRg7N/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781955746769051-7083-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task1JRg7N/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781955746769051-7083-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.6.234.254:44633
--webserver_interface=127.6.234.254
--webserver_port=39859
--keytab_file=/tmp/dist-test-task1JRg7N/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781955746769051-7083-0/krb5kdc/kudu_127.6.234.254.keytab
--principal=kudu/127.6.234.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.6.234.212:34463
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.6.234.254:44633
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--ipki_private_key_password_cmd=echo foo with env {KRB5CCNAME=/tmp/dist-test-task1JRg7N/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781955746769051-7083-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task1JRg7N/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781955746769051-7083-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task1JRg7N/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781955746769051-7083-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 11:44:58.513106 21306 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 11:44:58.513315 21306 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 11:44:58.513338 21306 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 11:44:58.514573 21306 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 11:44:58.514618 21306 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 11:44:58.514632 21306 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 11:44:58.514645 21306 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 11:44:58.514698 21306 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 11:44:58.516155 21306 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.6.234.212:34463
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task1JRg7N/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781955746769051-7083-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task1JRg7N/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781955746769051-7083-0/minicluster-data/master-0/wal
--ipki_private_key_password_cmd=echo foo
--ipki_ca_key_size=768
--master_addresses=127.6.234.254:44633
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task1JRg7N/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781955746769051-7083-0/krb5kdc/kudu_127.6.234.254.keytab
--principal=kudu/127.6.234.254
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.6.234.254:44633
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task1JRg7N/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781955746769051-7083-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.6.234.254
--webserver_port=39859
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.21306
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task1JRg7N/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781955746769051-7083-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 20 Jun 2026 10:32:37 UTC on ccycloud.quasar-bgyofv.root.comops.site
build id 4457173
I20260620 11:44:58.516404 21306 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 11:44:58.516662 21306 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 11:44:58.525373 21315 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 11:44:58.525389 21312 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 11:44:58.525393 21313 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 11:44:58.525905 21306 server_base.cc:1034] running on GCE node
I20260620 11:44:58.526129 21306 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 11:44:58.526392 21306 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 11:44:58.527530 21306 hybrid_clock.cc:648] HybridClock initialized: now 1781955898527516 us; error 26 us; skew 500 ppm
Jun 20 11:44:58 dist-test-slave-cjc6 krb5kdc[20762](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781955898, etypes {rep=17 tkt=17 ses=17}, kudu/127.6.234.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 11:44:58.536615 21306 init.cc:377] Logged in from keytab as kudu/127.6.234.254@KRBTEST.COM (short username kudu)
I20260620 11:44:58.537166 21306 webserver.cc:460] Webserver started at http://127.6.234.254:39859/ using document root <none> and password file <none>
I20260620 11:44:58.537428 21306 fs_manager.cc:359] Metadata directory not provided
I20260620 11:44:58.537518 21306 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 11:44:58.538987 21306 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.001s sys 0.000s
I20260620 11:44:58.539656 21322 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 11:44:58.539834 21306 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260620 11:44:58.539949 21306 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task1JRg7N/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781955746769051-7083-0/minicluster-data/master-0/data,/tmp/dist-test-task1JRg7N/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781955746769051-7083-0/minicluster-data/master-0/wal
uuid: "4195d863c1404c0a8137a88b6d8ad6f1"
format_stamp: "Formatted at 2026-06-20 11:44:57 on dist-test-slave-cjc6"
server_key: "6bc565ce9f35dadb384c5433eecd9a63"
server_key_iv: "9b7b41592b4da7fed09776909d1dedc9"
server_key_version: "encryptionkey@0"
I20260620 11:44:58.540354 21306 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task1JRg7N/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781955746769051-7083-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task1JRg7N/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781955746769051-7083-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task1JRg7N/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781955746769051-7083-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 11:44:58.552228 21306 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 11:44:58.554610 21306 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 11:44:58.554768 21306 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 11:44:58.558966 21306 rpc_server.cc:307] RPC server started. Bound to: 127.6.234.254:44633
I20260620 11:44:58.559000 21374 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.6.234.254:44633 every 8 connection(s)
I20260620 11:44:58.559417 21306 server_base.cc:1166] Dumped server information to /tmp/dist-test-task1JRg7N/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781955746769051-7083-0/minicluster-data/master-0/data/info.pb
I20260620 11:44:58.561033 21375 sys_catalog.cc:263] Verifying existing consensus state
I20260620 11:44:58.561620 21375 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 4195d863c1404c0a8137a88b6d8ad6f1: Bootstrap starting.
I20260620 11:44:58.568215 7083 external_mini_cluster.cc:1388] Started /tmp/dist-test-task1JRg7N/build/release/bin/kudu as pid 21306
I20260620 11:44:58.568576 7083 external_mini_cluster.cc:1326] Running /tmp/dist-test-task1JRg7N/build/release/bin/kudu
/tmp/dist-test-task1JRg7N/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task1JRg7N/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781955746769051-7083-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-task1JRg7N/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781955746769051-7083-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task1JRg7N/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781955746769051-7083-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-task1JRg7N/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781955746769051-7083-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.6.234.193:35733
--local_ip_for_outbound_sockets=127.6.234.193
--tserver_master_addrs=127.6.234.254:44633
--webserver_port=45589
--webserver_interface=127.6.234.193
--keytab_file=/tmp/dist-test-task1JRg7N/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781955746769051-7083-0/krb5kdc/kudu_127.6.234.193.keytab
--principal=kudu/127.6.234.193
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.6.234.212:34463
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task1JRg7N/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781955746769051-7083-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task1JRg7N/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781955746769051-7083-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task1JRg7N/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781955746769051-7083-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260620 11:44:58.570472 21375 log.cc:824] T 00000000000000000000000000000000 P 4195d863c1404c0a8137a88b6d8ad6f1: Log is configured to *not* fsync() on all Append() calls
I20260620 11:44:58.572620 21375 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 4195d863c1404c0a8137a88b6d8ad6f1: Bootstrap replayed 1/1 log segments. Stats: ops{read=8 overwritten=0 applied=8 ignored=0} inserts{seen=5 ignored=0} mutations{seen=4 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260620 11:44:58.572937 21375 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 4195d863c1404c0a8137a88b6d8ad6f1: Bootstrap complete.
I20260620 11:44:58.575246 21375 raft_consensus.cc:348] T 00000000000000000000000000000000 P 4195d863c1404c0a8137a88b6d8ad6f1 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4195d863c1404c0a8137a88b6d8ad6f1" member_type: VOTER last_known_addr { host: "127.6.234.254" port: 44633 } }
I20260620 11:44:58.575466 21375 raft_consensus.cc:729] T 00000000000000000000000000000000 P 4195d863c1404c0a8137a88b6d8ad6f1 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4195d863c1404c0a8137a88b6d8ad6f1, State: Initialized, Role: FOLLOWER
I20260620 11:44:58.575640 21375 consensus_queue.cc:260] T 00000000000000000000000000000000 P 4195d863c1404c0a8137a88b6d8ad6f1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 8, Last appended: 1.8, Last appended by leader: 8, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4195d863c1404c0a8137a88b6d8ad6f1" member_type: VOTER last_known_addr { host: "127.6.234.254" port: 44633 } }
I20260620 11:44:58.575708 21375 raft_consensus.cc:388] T 00000000000000000000000000000000 P 4195d863c1404c0a8137a88b6d8ad6f1 [term 1 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260620 11:44:58.575762 21375 raft_consensus.cc:482] T 00000000000000000000000000000000 P 4195d863c1404c0a8137a88b6d8ad6f1 [term 1 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260620 11:44:58.575810 21375 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 4195d863c1404c0a8137a88b6d8ad6f1 [term 1 FOLLOWER]: Advancing to term 2
I20260620 11:44:58.576817 21375 raft_consensus.cc:504] T 00000000000000000000000000000000 P 4195d863c1404c0a8137a88b6d8ad6f1 [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4195d863c1404c0a8137a88b6d8ad6f1" member_type: VOTER last_known_addr { host: "127.6.234.254" port: 44633 } }
I20260620 11:44:58.576905 21375 leader_election.cc:302] T 00000000000000000000000000000000 P 4195d863c1404c0a8137a88b6d8ad6f1 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 4195d863c1404c0a8137a88b6d8ad6f1; no voters:
I20260620 11:44:58.577090 21375 leader_election.cc:288] T 00000000000000000000000000000000 P 4195d863c1404c0a8137a88b6d8ad6f1 [CANDIDATE]: Term 2 election: Requested vote from peers
I20260620 11:44:58.577129 21379 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 4195d863c1404c0a8137a88b6d8ad6f1 [term 2 FOLLOWER]: Leader election won for term 2
I20260620 11:44:58.577299 21379 raft_consensus.cc:686] T 00000000000000000000000000000000 P 4195d863c1404c0a8137a88b6d8ad6f1 [term 2 LEADER]: Becoming Leader. State: Replica: 4195d863c1404c0a8137a88b6d8ad6f1, State: Running, Role: LEADER
I20260620 11:44:58.577374 21375 sys_catalog.cc:564] T 00000000000000000000000000000000 P 4195d863c1404c0a8137a88b6d8ad6f1 [sys.catalog]: configured and running, proceeding with master startup.
I20260620 11:44:58.577450 21379 consensus_queue.cc:237] T 00000000000000000000000000000000 P 4195d863c1404c0a8137a88b6d8ad6f1 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 1.8, Last appended by leader: 8, Current term: 2, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4195d863c1404c0a8137a88b6d8ad6f1" member_type: VOTER last_known_addr { host: "127.6.234.254" port: 44633 } }
I20260620 11:44:58.577761 21380 sys_catalog.cc:455] T 00000000000000000000000000000000 P 4195d863c1404c0a8137a88b6d8ad6f1 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 2 leader_uuid: "4195d863c1404c0a8137a88b6d8ad6f1" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4195d863c1404c0a8137a88b6d8ad6f1" member_type: VOTER last_known_addr { host: "127.6.234.254" port: 44633 } } }
I20260620 11:44:58.577872 21380 sys_catalog.cc:458] T 00000000000000000000000000000000 P 4195d863c1404c0a8137a88b6d8ad6f1 [sys.catalog]: This master's current role is: LEADER
I20260620 11:44:58.577759 21379 sys_catalog.cc:455] T 00000000000000000000000000000000 P 4195d863c1404c0a8137a88b6d8ad6f1 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 4195d863c1404c0a8137a88b6d8ad6f1. Latest consensus state: current_term: 2 leader_uuid: "4195d863c1404c0a8137a88b6d8ad6f1" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4195d863c1404c0a8137a88b6d8ad6f1" member_type: VOTER last_known_addr { host: "127.6.234.254" port: 44633 } } }
I20260620 11:44:58.578068 21379 sys_catalog.cc:458] T 00000000000000000000000000000000 P 4195d863c1404c0a8137a88b6d8ad6f1 [sys.catalog]: This master's current role is: LEADER
I20260620 11:44:58.578198 21386 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260620 11:44:58.578985 21386 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260620 11:44:58.579088 21386 catalog_manager.cc:1259] Loaded cluster ID: 19042a500bc94fdabc05bb1fbd473859
I20260620 11:44:58.579124 21386 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260620 11:44:58.579365 21386 catalog_manager.cc:1314] IPKI private key was stored in cleartext, attempting to encrypt it.
I20260620 11:44:58.584007 21386 catalog_manager.cc:1324] IPKI private key successfully encrypted.
I20260620 11:44:58.586478 21386 catalog_manager.cc:1504] Loading token signing keys...
I20260620 11:44:58.586683 21386 catalog_manager.cc:5879] T 00000000000000000000000000000000 P 4195d863c1404c0a8137a88b6d8ad6f1: Loaded TSK: 0
I20260620 11:44:58.586910 21386 catalog_manager.cc:1514] Initializing in-progress tserver states...
W20260620 11:44:58.651294 21377 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 11:44:58.651542 21377 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 11:44:58.651583 21377 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 11:44:58.652731 21377 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 11:44:58.652804 21377 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 11:44:58.652865 21377 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.6.234.193
I20260620 11:44:58.654428 21377 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.6.234.212:34463
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task1JRg7N/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781955746769051-7083-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-task1JRg7N/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781955746769051-7083-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task1JRg7N/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781955746769051-7083-0/krb5kdc/kudu_127.6.234.193.keytab
--principal=kudu/127.6.234.193
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.6.234.193:35733
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task1JRg7N/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781955746769051-7083-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.6.234.193
--webserver_port=45589
--tserver_master_addrs=127.6.234.254:44633
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.21377
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.6.234.193
--log_dir=/tmp/dist-test-task1JRg7N/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781955746769051-7083-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 20 Jun 2026 10:32:37 UTC on ccycloud.quasar-bgyofv.root.comops.site
build id 4457173
I20260620 11:44:58.654695 21377 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 11:44:58.654927 21377 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 11:44:58.660805 21402 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 11:44:58.660772 21403 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 11:44:58.661031 21377 server_base.cc:1034] running on GCE node
W20260620 11:44:58.660791 21405 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 11:44:58.661293 21377 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 11:44:58.661540 21377 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 11:44:58.662704 21377 hybrid_clock.cc:648] HybridClock initialized: now 1781955898662676 us; error 44 us; skew 500 ppm
Jun 20 11:44:58 dist-test-slave-cjc6 krb5kdc[20762](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781955898, etypes {rep=17 tkt=17 ses=17}, kudu/127.6.234.193@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 11:44:58.665299 21377 init.cc:377] Logged in from keytab as kudu/127.6.234.193@KRBTEST.COM (short username kudu)
I20260620 11:44:58.665779 21377 webserver.cc:460] Webserver started at http://127.6.234.193:45589/ using document root <none> and password file <none>
I20260620 11:44:58.666004 21377 fs_manager.cc:359] Metadata directory not provided
I20260620 11:44:58.666085 21377 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 11:44:58.667486 21377 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.000s sys 0.001s
I20260620 11:44:58.668200 21412 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 11:44:58.668375 21377 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.001s
I20260620 11:44:58.668471 21377 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task1JRg7N/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781955746769051-7083-0/minicluster-data/ts-0/data,/tmp/dist-test-task1JRg7N/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781955746769051-7083-0/minicluster-data/ts-0/wal
uuid: "6705ef55d9584b59865e2da234933258"
format_stamp: "Formatted at 2026-06-20 11:44:57 on dist-test-slave-cjc6"
server_key: "aaecbf620d1e90b359cbec9b916a3ad9"
server_key_iv: "36d40e1318d339ab95f1f04a89b23817"
server_key_version: "encryptionkey@0"
I20260620 11:44:58.668820 21377 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task1JRg7N/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781955746769051-7083-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-task1JRg7N/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781955746769051-7083-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-task1JRg7N/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781955746769051-7083-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 11:44:58.677196 21377 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 11:44:58.679637 21377 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 11:44:58.679780 21377 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 11:44:58.680006 21377 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 11:44:58.680546 21419 ts_tablet_manager.cc:536] Loading tablet metadata (0/1 complete)
I20260620 11:44:58.682480 21377 ts_tablet_manager.cc:579] Loaded tablet metadata (1 total tablets, 1 live tablets)
I20260620 11:44:58.682540 21377 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.002s user 0.000s sys 0.000s
I20260620 11:44:58.682585 21377 ts_tablet_manager.cc:594] Registering tablets (0/1 complete)
I20260620 11:44:58.683213 21377 ts_tablet_manager.cc:610] Registered 1 tablets
I20260620 11:44:58.683270 21377 ts_tablet_manager.cc:589] Time spent register tablets: real 0.001s user 0.000s sys 0.000s
I20260620 11:44:58.683337 21419 tablet_bootstrap.cc:492] T b5b6c99c41d54a56b52e4e7bbfe1cefa P 6705ef55d9584b59865e2da234933258: Bootstrap starting.
I20260620 11:44:58.690433 21377 rpc_server.cc:307] RPC server started. Bound to: 127.6.234.193:35733
I20260620 11:44:58.690476 21526 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.6.234.193:35733 every 8 connection(s)
I20260620 11:44:58.690879 21377 server_base.cc:1166] Dumped server information to /tmp/dist-test-task1JRg7N/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781955746769051-7083-0/minicluster-data/ts-0/data/info.pb
I20260620 11:44:58.694010 7083 external_mini_cluster.cc:1388] Started /tmp/dist-test-task1JRg7N/build/release/bin/kudu as pid 21377
I20260620 11:44:58.694337 7083 external_mini_cluster.cc:1326] Running /tmp/dist-test-task1JRg7N/build/release/bin/kudu
/tmp/dist-test-task1JRg7N/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task1JRg7N/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781955746769051-7083-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-task1JRg7N/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781955746769051-7083-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task1JRg7N/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781955746769051-7083-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-task1JRg7N/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781955746769051-7083-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.6.234.194:40017
--local_ip_for_outbound_sockets=127.6.234.194
--tserver_master_addrs=127.6.234.254:44633
--webserver_port=40019
--webserver_interface=127.6.234.194
--keytab_file=/tmp/dist-test-task1JRg7N/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781955746769051-7083-0/krb5kdc/kudu_127.6.234.194.keytab
--principal=kudu/127.6.234.194
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.6.234.212:34463
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task1JRg7N/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781955746769051-7083-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task1JRg7N/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781955746769051-7083-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task1JRg7N/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781955746769051-7083-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260620 11:44:58.694640 21419 log.cc:824] T b5b6c99c41d54a56b52e4e7bbfe1cefa P 6705ef55d9584b59865e2da234933258: Log is configured to *not* fsync() on all Append() calls
Jun 20 11:44:58 dist-test-slave-cjc6 krb5kdc[20762](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781955898, etypes {rep=17 tkt=17 ses=17}, kudu/127.6.234.193@KRBTEST.COM for kudu/127.6.234.254@KRBTEST.COM
I20260620 11:44:58.699527 21419 tablet_bootstrap.cc:492] T b5b6c99c41d54a56b52e4e7bbfe1cefa P 6705ef55d9584b59865e2da234933258: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=1 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260620 11:44:58.699874 21419 tablet_bootstrap.cc:492] T b5b6c99c41d54a56b52e4e7bbfe1cefa P 6705ef55d9584b59865e2da234933258: Bootstrap complete.
I20260620 11:44:58.701140 21419 ts_tablet_manager.cc:1397] T b5b6c99c41d54a56b52e4e7bbfe1cefa P 6705ef55d9584b59865e2da234933258: Time spent bootstrapping tablet: real 0.018s user 0.005s sys 0.006s
I20260620 11:44:58.702584 21419 raft_consensus.cc:348] T b5b6c99c41d54a56b52e4e7bbfe1cefa P 6705ef55d9584b59865e2da234933258 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fd553ed4672d4f84a1058e83d69d0e88" member_type: VOTER last_known_addr { host: "127.6.234.194" port: 40017 } } peers { permanent_uuid: "6705ef55d9584b59865e2da234933258" member_type: VOTER last_known_addr { host: "127.6.234.193" port: 35733 } } peers { permanent_uuid: "741432935b6d42c38ab97037b7ab9b6f" member_type: VOTER last_known_addr { host: "127.6.234.195" port: 44517 } }
I20260620 11:44:58.702945 21419 raft_consensus.cc:729] T b5b6c99c41d54a56b52e4e7bbfe1cefa P 6705ef55d9584b59865e2da234933258 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6705ef55d9584b59865e2da234933258, State: Initialized, Role: FOLLOWER
I20260620 11:44:58.703083 21419 consensus_queue.cc:260] T b5b6c99c41d54a56b52e4e7bbfe1cefa P 6705ef55d9584b59865e2da234933258 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fd553ed4672d4f84a1058e83d69d0e88" member_type: VOTER last_known_addr { host: "127.6.234.194" port: 40017 } } peers { permanent_uuid: "6705ef55d9584b59865e2da234933258" member_type: VOTER last_known_addr { host: "127.6.234.193" port: 35733 } } peers { permanent_uuid: "741432935b6d42c38ab97037b7ab9b6f" member_type: VOTER last_known_addr { host: "127.6.234.195" port: 44517 } }
I20260620 11:44:58.703357 21419 ts_tablet_manager.cc:1428] T b5b6c99c41d54a56b52e4e7bbfe1cefa P 6705ef55d9584b59865e2da234933258: Time spent starting tablet: real 0.002s user 0.006s sys 0.000s
I20260620 11:44:58.704094 21529 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 11:44:58.691680 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.234.254:44633 (local address 127.6.234.193:42135)
0620 11:44:58.691870 (+ 190us) negotiation.cc:107] Waiting for socket to connect
0620 11:44:58.691905 (+ 35us) client_negotiation.cc:174] Beginning negotiation
0620 11:44:58.692470 (+ 565us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 11:44:58.692985 (+ 515us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 11:44:58.692989 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 11:44:58.693269 (+ 280us) client_negotiation.cc:189] Negotiated authn=SASL
0620 11:44:58.693456 (+ 187us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 11:44:58.693461 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 11:44:58.695580 (+ 2119us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 11:44:58.695581 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 11:44:58.696485 (+ 904us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 11:44:58.696488 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 11:44:58.697069 (+ 581us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 11:44:58.697071 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 11:44:58.697200 (+ 129us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 11:44:58.697836 (+ 636us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 11:44:58.697847 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0620 11:44:58.699749 (+ 1902us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 11:44:58.702131 (+ 2382us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 11:44:58.702136 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 11:44:58.702138 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0620 11:44:58.702433 (+ 295us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 11:44:58.702725 (+ 292us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 11:44:58.702726 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 11:44:58.702726 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0620 11:44:58.702845 (+ 119us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 11:44:58.703481 (+ 636us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 11:44:58.703487 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 11:44:58.703605 (+ 118us) client_negotiation.cc:769] Sending connection context
0620 11:44:58.703724 (+ 119us) client_negotiation.cc:240] Negotiation successful
0620 11:44:58.703897 (+ 173us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":148,"thread_start_us":107,"threads_started":1}
I20260620 11:44:58.704106 21530 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 11:44:58.692071 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.6.234.193:42135 (local address 127.6.234.254:44633)
0620 11:44:58.692637 (+ 566us) server_negotiation.cc:206] Beginning negotiation
0620 11:44:58.692645 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0620 11:44:58.692672 (+ 27us) server_negotiation.cc:407] Connection header received
0620 11:44:58.692773 (+ 101us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 11:44:58.692777 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 11:44:58.692817 (+ 40us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 11:44:58.692957 (+ 140us) server_negotiation.cc:226] Negotiated authn=SASL
0620 11:44:58.693579 (+ 622us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 11:44:58.695443 (+ 1864us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 11:44:58.696582 (+ 1139us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 11:44:58.696961 (+ 379us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 11:44:58.697083 (+ 122us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 11:44:58.699867 (+ 2784us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 11:44:58.699871 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 11:44:58.699872 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 11:44:58.699884 (+ 12us) server_negotiation.cc:922] calling sasl_server_start()
0620 11:44:58.701960 (+ 2076us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 11:44:58.702531 (+ 571us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 11:44:58.702534 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 11:44:58.702534 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 11:44:58.702648 (+ 114us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 11:44:58.702920 (+ 272us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 11:44:58.702921 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 11:44:58.702921 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 11:44:58.703209 (+ 288us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 11:44:58.703292 (+ 83us) server_negotiation.cc:1035] Waiting for connection context
0620 11:44:58.703793 (+ 501us) server_negotiation.cc:299] Negotiation successful
0620 11:44:58.703930 (+ 137us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":526,"thread_start_us":83,"threads_started":1}
I20260620 11:44:58.705466 21527 heartbeater.cc:347] Connected to a master server at 127.6.234.254:44633
I20260620 11:44:58.705577 21527 heartbeater.cc:464] Registering TS with master...
I20260620 11:44:58.705793 21527 heartbeater.cc:511] Master 127.6.234.254:44633 requested a full tablet report, sending...
I20260620 11:44:58.706512 21339 ts_manager.cc:194] Registered new tserver with Master: 6705ef55d9584b59865e2da234933258 (127.6.234.193:35733)
I20260620 11:44:58.707571 21339 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.6.234.193@KRBTEST.COM'} at 127.6.234.193:42135
I20260620 11:44:58.708384 21527 heartbeater.cc:503] Master 127.6.234.254:44633 was elected leader, sending a full tablet report...
I20260620 11:44:58.713575 21534 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 11:44:58.707091 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.234.193:35733 (local address 127.0.0.1:52752)
0620 11:44:58.709930 (+ 2839us) negotiation.cc:107] Waiting for socket to connect
0620 11:44:58.709963 (+ 33us) client_negotiation.cc:174] Beginning negotiation
0620 11:44:58.710070 (+ 107us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 11:44:58.710324 (+ 254us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 11:44:58.710327 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 11:44:58.710329 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 11:44:58.710432 (+ 103us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 11:44:58.710434 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 11:44:58.711335 (+ 901us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 11:44:58.711337 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 11:44:58.712261 (+ 924us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 11:44:58.712264 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 11:44:58.713218 (+ 954us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 11:44:58.713219 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 11:44:58.713319 (+ 100us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 11:44:58.713342 (+ 23us) client_negotiation.cc:769] Sending connection context
0620 11:44:58.713440 (+ 98us) client_negotiation.cc:240] Negotiation successful
0620 11:44:58.713499 (+ 59us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":2812,"thread_start_us":96,"threads_started":1}
I20260620 11:44:58.713574 21535 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 11:44:58.707345 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52752 (local address 127.6.234.193:35733)
0620 11:44:58.709862 (+ 2517us) server_negotiation.cc:206] Beginning negotiation
0620 11:44:58.709866 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 11:44:58.710047 (+ 181us) server_negotiation.cc:407] Connection header received
0620 11:44:58.710192 (+ 145us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 11:44:58.710194 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 11:44:58.710210 (+ 16us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 11:44:58.710305 (+ 95us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 11:44:58.710520 (+ 215us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 11:44:58.711241 (+ 721us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 11:44:58.712363 (+ 1122us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 11:44:58.713096 (+ 733us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 11:44:58.713214 (+ 118us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 11:44:58.713274 (+ 60us) server_negotiation.cc:1035] Waiting for connection context
0620 11:44:58.713461 (+ 187us) server_negotiation.cc:299] Negotiation successful
0620 11:44:58.713501 (+ 40us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":2484,"thread_start_us":78,"threads_started":1}
I20260620 11:44:58.714309 21445 tablet_service.cc:1514] Processing DeleteTablet for tablet b5b6c99c41d54a56b52e4e7bbfe1cefa with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 11:44:58 UTC) from {username='kudu', principal='kudu/127.6.234.254@KRBTEST.COM'} at 127.0.0.1:52752
I20260620 11:44:58.714807 21536 tablet_replica.cc:330] stopping tablet replica
I20260620 11:44:58.714942 21536 raft_consensus.cc:2229] T b5b6c99c41d54a56b52e4e7bbfe1cefa P 6705ef55d9584b59865e2da234933258 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260620 11:44:58.715014 21536 raft_consensus.cc:2258] T b5b6c99c41d54a56b52e4e7bbfe1cefa P 6705ef55d9584b59865e2da234933258 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 11:44:58.715327 21536 ts_tablet_manager.cc:1905] T b5b6c99c41d54a56b52e4e7bbfe1cefa P 6705ef55d9584b59865e2da234933258: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 11:44:58.717321 21536 ts_tablet_manager.cc:1918] T b5b6c99c41d54a56b52e4e7bbfe1cefa P 6705ef55d9584b59865e2da234933258: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260620 11:44:58.717516 21536 log.cc:1196] T b5b6c99c41d54a56b52e4e7bbfe1cefa P 6705ef55d9584b59865e2da234933258: Deleting WAL directory at /tmp/dist-test-task1JRg7N/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781955746769051-7083-0/minicluster-data/ts-0/wal/wals/b5b6c99c41d54a56b52e4e7bbfe1cefa
I20260620 11:44:58.717856 21536 ts_tablet_manager.cc:1939] T b5b6c99c41d54a56b52e4e7bbfe1cefa P 6705ef55d9584b59865e2da234933258: Deleting consensus metadata
I20260620 11:44:58.718382 21326 catalog_manager.cc:4841] TS 6705ef55d9584b59865e2da234933258 (127.6.234.193:35733): tablet b5b6c99c41d54a56b52e4e7bbfe1cefa (table test-table [id=70697314689944c496e80b022368cd0d]) successfully deleted
W20260620 11:44:58.784955 21532 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 11:44:58.785177 21532 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 11:44:58.785202 21532 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 11:44:58.786289 21532 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 11:44:58.786324 21532 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 11:44:58.786350 21532 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.6.234.194
I20260620 11:44:58.787681 21532 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.6.234.212:34463
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task1JRg7N/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781955746769051-7083-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-task1JRg7N/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781955746769051-7083-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task1JRg7N/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781955746769051-7083-0/krb5kdc/kudu_127.6.234.194.keytab
--principal=kudu/127.6.234.194
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.6.234.194:40017
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task1JRg7N/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781955746769051-7083-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.6.234.194
--webserver_port=40019
--tserver_master_addrs=127.6.234.254:44633
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.21532
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.6.234.194
--log_dir=/tmp/dist-test-task1JRg7N/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781955746769051-7083-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 20 Jun 2026 10:32:37 UTC on ccycloud.quasar-bgyofv.root.comops.site
build id 4457173
I20260620 11:44:58.787856 21532 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 11:44:58.788028 21532 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 11:44:58.793145 21542 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 11:44:58.793154 21541 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 11:44:58.793256 21532 server_base.cc:1034] running on GCE node
W20260620 11:44:58.793154 21544 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 11:44:58.793608 21532 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 11:44:58.793896 21532 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 11:44:58.795092 21532 hybrid_clock.cc:648] HybridClock initialized: now 1781955898795087 us; error 41 us; skew 500 ppm
Jun 20 11:44:58 dist-test-slave-cjc6 krb5kdc[20762](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781955898, etypes {rep=17 tkt=17 ses=17}, kudu/127.6.234.194@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 11:44:58.797835 21532 init.cc:377] Logged in from keytab as kudu/127.6.234.194@KRBTEST.COM (short username kudu)
I20260620 11:44:58.798359 21532 webserver.cc:460] Webserver started at http://127.6.234.194:40019/ using document root <none> and password file <none>
I20260620 11:44:58.798606 21532 fs_manager.cc:359] Metadata directory not provided
I20260620 11:44:58.798699 21532 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 11:44:58.800179 21532 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.002s sys 0.000s
I20260620 11:44:58.800829 21551 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 11:44:58.801007 21532 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260620 11:44:58.801124 21532 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task1JRg7N/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781955746769051-7083-0/minicluster-data/ts-1/data,/tmp/dist-test-task1JRg7N/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781955746769051-7083-0/minicluster-data/ts-1/wal
uuid: "fd553ed4672d4f84a1058e83d69d0e88"
format_stamp: "Formatted at 2026-06-20 11:44:57 on dist-test-slave-cjc6"
server_key: "175cb27c7fe999d5d8de3c5eb757ab3a"
server_key_iv: "7a8318dc18ca4f5755ecaf130d21ff5d"
server_key_version: "encryptionkey@0"
I20260620 11:44:58.801496 21532 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task1JRg7N/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781955746769051-7083-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-task1JRg7N/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781955746769051-7083-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-task1JRg7N/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781955746769051-7083-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 11:44:58.821975 21532 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 11:44:58.824023 21532 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 11:44:58.824213 21532 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 11:44:58.824481 21532 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 11:44:58.824820 21532 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 11:44:58.824882 21532 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 11:44:58.824930 21532 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 11:44:58.824958 21532 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 11:44:58.831576 21532 rpc_server.cc:307] RPC server started. Bound to: 127.6.234.194:40017
I20260620 11:44:58.831617 21664 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.6.234.194:40017 every 8 connection(s)
I20260620 11:44:58.832026 21532 server_base.cc:1166] Dumped server information to /tmp/dist-test-task1JRg7N/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781955746769051-7083-0/minicluster-data/ts-1/data/info.pb
Jun 20 11:44:58 dist-test-slave-cjc6 krb5kdc[20762](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781955898, etypes {rep=17 tkt=17 ses=17}, kudu/127.6.234.194@KRBTEST.COM for kudu/127.6.234.254@KRBTEST.COM
I20260620 11:44:58.841266 21530 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 11:44:58.832848 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.6.234.194:33719 (local address 127.6.234.254:44633)
0620 11:44:58.832931 (+ 83us) server_negotiation.cc:206] Beginning negotiation
0620 11:44:58.832933 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 11:44:58.833158 (+ 225us) server_negotiation.cc:407] Connection header received
0620 11:44:58.833718 (+ 560us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 11:44:58.833719 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 11:44:58.833729 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 11:44:58.833825 (+ 96us) server_negotiation.cc:226] Negotiated authn=SASL
0620 11:44:58.834820 (+ 995us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 11:44:58.835311 (+ 491us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 11:44:58.836392 (+ 1081us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 11:44:58.836605 (+ 213us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 11:44:58.836660 (+ 55us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 11:44:58.838894 (+ 2234us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 11:44:58.838895 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 11:44:58.838895 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 11:44:58.838901 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0620 11:44:58.840061 (+ 1160us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 11:44:58.840423 (+ 362us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 11:44:58.840424 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 11:44:58.840424 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 11:44:58.840473 (+ 49us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 11:44:58.840674 (+ 201us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 11:44:58.840675 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 11:44:58.840675 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 11:44:58.840772 (+ 97us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 11:44:58.840821 (+ 49us) server_negotiation.cc:1035] Waiting for connection context
0620 11:44:58.841088 (+ 267us) server_negotiation.cc:299] Negotiation successful
0620 11:44:58.841179 (+ 91us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":57}
I20260620 11:44:58.841372 7083 external_mini_cluster.cc:1388] Started /tmp/dist-test-task1JRg7N/build/release/bin/kudu as pid 21532
I20260620 11:44:58.841421 21667 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 11:44:58.832816 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.234.254:44633 (local address 127.6.234.194:33719)
0620 11:44:58.833010 (+ 194us) negotiation.cc:107] Waiting for socket to connect
0620 11:44:58.833056 (+ 46us) client_negotiation.cc:174] Beginning negotiation
0620 11:44:58.833594 (+ 538us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 11:44:58.833892 (+ 298us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 11:44:58.833895 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 11:44:58.834513 (+ 618us) client_negotiation.cc:189] Negotiated authn=SASL
0620 11:44:58.834724 (+ 211us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 11:44:58.834729 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 11:44:58.835408 (+ 679us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 11:44:58.835409 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 11:44:58.836311 (+ 902us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 11:44:58.836314 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 11:44:58.836715 (+ 401us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 11:44:58.836716 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 11:44:58.836828 (+ 112us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 11:44:58.837355 (+ 527us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 11:44:58.837364 (+ 9us) client_negotiation.cc:656] calling sasl_client_start()
0620 11:44:58.838808 (+ 1444us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 11:44:58.840134 (+ 1326us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 11:44:58.840139 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 11:44:58.840141 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0620 11:44:58.840358 (+ 217us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 11:44:58.840529 (+ 171us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 11:44:58.840530 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 11:44:58.840530 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0620 11:44:58.840622 (+ 92us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 11:44:58.840825 (+ 203us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 11:44:58.840829 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 11:44:58.840941 (+ 112us) client_negotiation.cc:769] Sending connection context
0620 11:44:58.841024 (+ 83us) client_negotiation.cc:240] Negotiation successful
0620 11:44:58.841136 (+ 112us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":150,"thread_start_us":120,"threads_started":1}
I20260620 11:44:58.841670 7083 external_mini_cluster.cc:1326] Running /tmp/dist-test-task1JRg7N/build/release/bin/kudu
/tmp/dist-test-task1JRg7N/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task1JRg7N/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781955746769051-7083-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-task1JRg7N/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781955746769051-7083-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task1JRg7N/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781955746769051-7083-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-task1JRg7N/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781955746769051-7083-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.6.234.195:44517
--local_ip_for_outbound_sockets=127.6.234.195
--tserver_master_addrs=127.6.234.254:44633
--webserver_port=35283
--webserver_interface=127.6.234.195
--keytab_file=/tmp/dist-test-task1JRg7N/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781955746769051-7083-0/krb5kdc/kudu_127.6.234.195.keytab
--principal=kudu/127.6.234.195
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.6.234.212:34463
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task1JRg7N/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781955746769051-7083-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task1JRg7N/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781955746769051-7083-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task1JRg7N/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781955746769051-7083-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260620 11:44:58.841995 21665 heartbeater.cc:347] Connected to a master server at 127.6.234.254:44633
I20260620 11:44:58.842101 21665 heartbeater.cc:464] Registering TS with master...
I20260620 11:44:58.842289 21665 heartbeater.cc:511] Master 127.6.234.254:44633 requested a full tablet report, sending...
I20260620 11:44:58.842700 21339 ts_manager.cc:194] Registered new tserver with Master: fd553ed4672d4f84a1058e83d69d0e88 (127.6.234.194:40017)
I20260620 11:44:58.843109 21339 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.6.234.194@KRBTEST.COM'} at 127.6.234.194:33719
W20260620 11:44:58.922996 21668 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 11:44:58.923187 21668 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 11:44:58.923210 21668 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 11:44:58.924249 21668 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 11:44:58.924285 21668 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 11:44:58.924310 21668 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.6.234.195
I20260620 11:44:58.925631 21668 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.6.234.212:34463
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task1JRg7N/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781955746769051-7083-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-task1JRg7N/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781955746769051-7083-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task1JRg7N/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781955746769051-7083-0/krb5kdc/kudu_127.6.234.195.keytab
--principal=kudu/127.6.234.195
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.6.234.195:44517
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task1JRg7N/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781955746769051-7083-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.6.234.195
--webserver_port=35283
--tserver_master_addrs=127.6.234.254:44633
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.21668
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.6.234.195
--log_dir=/tmp/dist-test-task1JRg7N/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781955746769051-7083-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 20 Jun 2026 10:32:37 UTC on ccycloud.quasar-bgyofv.root.comops.site
build id 4457173
I20260620 11:44:58.925872 21668 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 11:44:58.926049 21668 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 11:44:58.930830 21676 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 11:44:58.930867 21673 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 11:44:58.930900 21674 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 11:44:58.931123 21668 server_base.cc:1034] running on GCE node
I20260620 11:44:58.931336 21668 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 11:44:58.931568 21668 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 11:44:58.932725 21668 hybrid_clock.cc:648] HybridClock initialized: now 1781955898932708 us; error 28 us; skew 500 ppm
Jun 20 11:44:58 dist-test-slave-cjc6 krb5kdc[20762](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781955898, etypes {rep=17 tkt=17 ses=17}, kudu/127.6.234.195@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 11:44:58.935168 21668 init.cc:377] Logged in from keytab as kudu/127.6.234.195@KRBTEST.COM (short username kudu)
I20260620 11:44:58.935590 21668 webserver.cc:460] Webserver started at http://127.6.234.195:35283/ using document root <none> and password file <none>
I20260620 11:44:58.935822 21668 fs_manager.cc:359] Metadata directory not provided
I20260620 11:44:58.935904 21668 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 11:44:58.937176 21668 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.001s sys 0.000s
I20260620 11:44:58.937877 21683 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 11:44:58.938079 21668 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260620 11:44:58.938177 21668 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task1JRg7N/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781955746769051-7083-0/minicluster-data/ts-2/data,/tmp/dist-test-task1JRg7N/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781955746769051-7083-0/minicluster-data/ts-2/wal
uuid: "741432935b6d42c38ab97037b7ab9b6f"
format_stamp: "Formatted at 2026-06-20 11:44:58 on dist-test-slave-cjc6"
server_key: "0fdbd66a6280cdebd7c4853a2e880ec0"
server_key_iv: "3a8b79d819f0fbd10c123d784ae2fdaf"
server_key_version: "encryptionkey@0"
I20260620 11:44:58.938515 21668 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task1JRg7N/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781955746769051-7083-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-task1JRg7N/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781955746769051-7083-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-task1JRg7N/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781955746769051-7083-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 11:44:58.977324 21668 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 11:44:58.979857 21668 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 11:44:58.979974 21668 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 11:44:58.980150 21668 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 11:44:58.980594 21690 ts_tablet_manager.cc:536] Loading tablet metadata (0/1 complete)
I20260620 11:44:58.982039 21668 ts_tablet_manager.cc:579] Loaded tablet metadata (1 total tablets, 1 live tablets)
I20260620 11:44:58.982082 21668 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.002s user 0.000s sys 0.000s
I20260620 11:44:58.982107 21668 ts_tablet_manager.cc:594] Registering tablets (0/1 complete)
I20260620 11:44:58.982758 21668 ts_tablet_manager.cc:610] Registered 1 tablets
I20260620 11:44:58.982792 21668 ts_tablet_manager.cc:589] Time spent register tablets: real 0.001s user 0.000s sys 0.000s
I20260620 11:44:58.982888 21690 tablet_bootstrap.cc:492] T b5b6c99c41d54a56b52e4e7bbfe1cefa P 741432935b6d42c38ab97037b7ab9b6f: Bootstrap starting.
I20260620 11:44:58.989441 21668 rpc_server.cc:307] RPC server started. Bound to: 127.6.234.195:44517
I20260620 11:44:58.989849 21797 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.6.234.195:44517 every 8 connection(s)
I20260620 11:44:58.989892 21668 server_base.cc:1166] Dumped server information to /tmp/dist-test-task1JRg7N/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781955746769051-7083-0/minicluster-data/ts-2/data/info.pb
I20260620 11:44:58.993976 21690 log.cc:824] T b5b6c99c41d54a56b52e4e7bbfe1cefa P 741432935b6d42c38ab97037b7ab9b6f: Log is configured to *not* fsync() on all Append() calls
I20260620 11:44:58.996546 21690 tablet_bootstrap.cc:492] T b5b6c99c41d54a56b52e4e7bbfe1cefa P 741432935b6d42c38ab97037b7ab9b6f: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=1 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260620 11:44:58.996866 21690 tablet_bootstrap.cc:492] T b5b6c99c41d54a56b52e4e7bbfe1cefa P 741432935b6d42c38ab97037b7ab9b6f: Bootstrap complete.
Jun 20 11:44:58 dist-test-slave-cjc6 krb5kdc[20762](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781955898, etypes {rep=17 tkt=17 ses=17}, kudu/127.6.234.195@KRBTEST.COM for kudu/127.6.234.254@KRBTEST.COM
I20260620 11:44:58.997418 7083 external_mini_cluster.cc:1388] Started /tmp/dist-test-task1JRg7N/build/release/bin/kudu as pid 21668
I20260620 11:44:58.997978 21690 ts_tablet_manager.cc:1397] T b5b6c99c41d54a56b52e4e7bbfe1cefa P 741432935b6d42c38ab97037b7ab9b6f: Time spent bootstrapping tablet: real 0.015s user 0.003s sys 0.011s
I20260620 11:44:58.999449 21690 raft_consensus.cc:348] T b5b6c99c41d54a56b52e4e7bbfe1cefa P 741432935b6d42c38ab97037b7ab9b6f [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fd553ed4672d4f84a1058e83d69d0e88" member_type: VOTER last_known_addr { host: "127.6.234.194" port: 40017 } } peers { permanent_uuid: "6705ef55d9584b59865e2da234933258" member_type: VOTER last_known_addr { host: "127.6.234.193" port: 35733 } } peers { permanent_uuid: "741432935b6d42c38ab97037b7ab9b6f" member_type: VOTER last_known_addr { host: "127.6.234.195" port: 44517 } }
I20260620 11:44:58.999799 21690 raft_consensus.cc:729] T b5b6c99c41d54a56b52e4e7bbfe1cefa P 741432935b6d42c38ab97037b7ab9b6f [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 741432935b6d42c38ab97037b7ab9b6f, State: Initialized, Role: FOLLOWER
I20260620 11:44:58.999969 21690 consensus_queue.cc:260] T b5b6c99c41d54a56b52e4e7bbfe1cefa P 741432935b6d42c38ab97037b7ab9b6f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fd553ed4672d4f84a1058e83d69d0e88" member_type: VOTER last_known_addr { host: "127.6.234.194" port: 40017 } } peers { permanent_uuid: "6705ef55d9584b59865e2da234933258" member_type: VOTER last_known_addr { host: "127.6.234.193" port: 35733 } } peers { permanent_uuid: "741432935b6d42c38ab97037b7ab9b6f" member_type: VOTER last_known_addr { host: "127.6.234.195" port: 44517 } }
I20260620 11:44:59.000257 21690 ts_tablet_manager.cc:1428] T b5b6c99c41d54a56b52e4e7bbfe1cefa P 741432935b6d42c38ab97037b7ab9b6f: Time spent starting tablet: real 0.002s user 0.000s sys 0.002s
I20260620 11:44:59.000943 21530 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 11:44:58.990753 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.6.234.195:60865 (local address 127.6.234.254:44633)
0620 11:44:58.990857 (+ 104us) server_negotiation.cc:206] Beginning negotiation
0620 11:44:58.990864 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0620 11:44:58.991109 (+ 245us) server_negotiation.cc:407] Connection header received
0620 11:44:58.991927 (+ 818us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 11:44:58.991928 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 11:44:58.991938 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 11:44:58.991981 (+ 43us) server_negotiation.cc:226] Negotiated authn=SASL
0620 11:44:58.992960 (+ 979us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 11:44:58.993435 (+ 475us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 11:44:58.994851 (+ 1416us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 11:44:58.995039 (+ 188us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 11:44:58.995097 (+ 58us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 11:44:58.997664 (+ 2567us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 11:44:58.997666 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 11:44:58.997667 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 11:44:58.997674 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0620 11:44:58.999427 (+ 1753us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 11:44:58.999907 (+ 480us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 11:44:58.999909 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 11:44:58.999909 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 11:44:58.999969 (+ 60us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 11:44:59.000247 (+ 278us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 11:44:59.000248 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 11:44:59.000249 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 11:44:59.000388 (+ 139us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 11:44:59.000472 (+ 84us) server_negotiation.cc:1035] Waiting for connection context
0620 11:44:59.000763 (+ 291us) server_negotiation.cc:299] Negotiation successful
0620 11:44:59.000851 (+ 88us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":43}
I20260620 11:44:59.001109 21800 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 11:44:58.990712 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.234.254:44633 (local address 127.6.234.195:60865)
0620 11:44:58.990960 (+ 248us) negotiation.cc:107] Waiting for socket to connect
0620 11:44:58.991009 (+ 49us) client_negotiation.cc:174] Beginning negotiation
0620 11:44:58.991818 (+ 809us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 11:44:58.992048 (+ 230us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 11:44:58.992055 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 11:44:58.992548 (+ 493us) client_negotiation.cc:189] Negotiated authn=SASL
0620 11:44:58.992868 (+ 320us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 11:44:58.992875 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 11:44:58.993551 (+ 676us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 11:44:58.993552 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 11:44:58.994756 (+ 1204us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 11:44:58.994759 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 11:44:58.995141 (+ 382us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 11:44:58.995143 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 11:44:58.995281 (+ 138us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 11:44:58.995779 (+ 498us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 11:44:58.995789 (+ 10us) client_negotiation.cc:656] calling sasl_client_start()
0620 11:44:58.997540 (+ 1751us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 11:44:58.999524 (+ 1984us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 11:44:58.999529 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 11:44:58.999531 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0620 11:44:58.999819 (+ 288us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 11:44:59.000044 (+ 225us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 11:44:59.000045 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 11:44:59.000045 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0620 11:44:59.000183 (+ 138us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 11:44:59.000465 (+ 282us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 11:44:59.000472 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 11:44:59.000623 (+ 151us) client_negotiation.cc:769] Sending connection context
0620 11:44:59.000708 (+ 85us) client_negotiation.cc:240] Negotiation successful
0620 11:44:59.000869 (+ 161us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":201,"thread_start_us":146,"threads_started":1}
I20260620 11:44:59.001757 21798 heartbeater.cc:347] Connected to a master server at 127.6.234.254:44633
I20260620 11:44:59.001852 21798 heartbeater.cc:464] Registering TS with master...
I20260620 11:44:59.002035 21798 heartbeater.cc:511] Master 127.6.234.254:44633 requested a full tablet report, sending...
I20260620 11:44:59.002592 21339 ts_manager.cc:194] Registered new tserver with Master: 741432935b6d42c38ab97037b7ab9b6f (127.6.234.195:44517)
I20260620 11:44:59.003345 21339 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.6.234.195@KRBTEST.COM'} at 127.6.234.195:60865
I20260620 11:44:59.004181 21798 heartbeater.cc:503] Master 127.6.234.254:44633 was elected leader, sending a full tablet report...
I20260620 11:44:59.005873 21803 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 11:44:58.997981 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52302 (local address 127.6.234.254:44633)
0620 11:44:58.998186 (+ 205us) server_negotiation.cc:206] Beginning negotiation
0620 11:44:58.998188 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 11:44:58.998195 (+ 7us) server_negotiation.cc:407] Connection header received
0620 11:44:58.998216 (+ 21us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 11:44:58.998217 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 11:44:58.998228 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 11:44:58.998292 (+ 64us) server_negotiation.cc:226] Negotiated authn=SASL
0620 11:44:58.999026 (+ 734us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 11:44:58.999511 (+ 485us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 11:44:59.000525 (+ 1014us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 11:44:59.000747 (+ 222us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 11:44:59.000820 (+ 73us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 11:44:59.002394 (+ 1574us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 11:44:59.002396 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 11:44:59.002396 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 11:44:59.002404 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0620 11:44:59.004603 (+ 2199us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 11:44:59.005016 (+ 413us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 11:44:59.005017 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 11:44:59.005018 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 11:44:59.005082 (+ 64us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 11:44:59.005293 (+ 211us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 11:44:59.005294 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 11:44:59.005295 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 11:44:59.005409 (+ 114us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 11:44:59.005568 (+ 159us) server_negotiation.cc:1035] Waiting for connection context
0620 11:44:59.005609 (+ 41us) server_negotiation.cc:299] Negotiation successful
0620 11:44:59.005766 (+ 157us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":184,"thread_start_us":53,"threads_started":1}
I20260620 11:44:59.006661 7083 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
Jun 20 11:44:59 dist-test-slave-cjc6 krb5kdc[20762](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781955898, etypes {rep=17 tkt=17 ses=17}, kudu/127.6.234.254@KRBTEST.COM for kudu/127.6.234.195@KRBTEST.COM
I20260620 11:44:59.009879 21803 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 11:44:59.007003 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52318 (local address 127.6.234.254:44633)
0620 11:44:59.007041 (+ 38us) server_negotiation.cc:206] Beginning negotiation
0620 11:44:59.007043 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 11:44:59.007166 (+ 123us) server_negotiation.cc:407] Connection header received
0620 11:44:59.007217 (+ 51us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 11:44:59.007218 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 11:44:59.007245 (+ 27us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 11:44:59.007317 (+ 72us) server_negotiation.cc:226] Negotiated authn=TOKEN
0620 11:44:59.007386 (+ 69us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 11:44:59.007992 (+ 606us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 11:44:59.008961 (+ 969us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 11:44:59.009290 (+ 329us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 11:44:59.009410 (+ 120us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 11:44:59.009517 (+ 107us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0620 11:44:59.009629 (+ 112us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0620 11:44:59.009754 (+ 125us) server_negotiation.cc:1035] Waiting for connection context
0620 11:44:59.009779 (+ 25us) server_negotiation.cc:299] Negotiation successful
0620 11:44:59.009806 (+ 27us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":20}
I20260620 11:44:59.011098 21534 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 11:44:59.002904 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.234.195:44517 (local address 127.0.0.1:48664)
0620 11:44:59.002942 (+ 38us) negotiation.cc:107] Waiting for socket to connect
0620 11:44:59.002954 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0620 11:44:59.002992 (+ 38us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 11:44:59.003407 (+ 415us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 11:44:59.003408 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 11:44:59.003818 (+ 410us) client_negotiation.cc:189] Negotiated authn=SASL
0620 11:44:59.003863 (+ 45us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 11:44:59.003865 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 11:44:59.004860 (+ 995us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 11:44:59.004860 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 11:44:59.005317 (+ 457us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 11:44:59.005318 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 11:44:59.005886 (+ 568us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 11:44:59.005887 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 11:44:59.005956 (+ 69us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 11:44:59.006446 (+ 490us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 11:44:59.006451 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0620 11:44:59.008071 (+ 1620us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 11:44:59.010103 (+ 2032us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 11:44:59.010109 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 11:44:59.010111 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0620 11:44:59.010325 (+ 214us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 11:44:59.010541 (+ 216us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 11:44:59.010541 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 11:44:59.010541 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0620 11:44:59.010592 (+ 51us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 11:44:59.010880 (+ 288us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 11:44:59.010885 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 11:44:59.010922 (+ 37us) client_negotiation.cc:769] Sending connection context
0620 11:44:59.010973 (+ 51us) client_negotiation.cc:240] Negotiation successful
0620 11:44:59.011041 (+ 68us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":19}
I20260620 11:44:59.011173 21805 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 11:44:59.003021 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48664 (local address 127.6.234.195:44517)
0620 11:44:59.003212 (+ 191us) server_negotiation.cc:206] Beginning negotiation
0620 11:44:59.003217 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 11:44:59.003229 (+ 12us) server_negotiation.cc:407] Connection header received
0620 11:44:59.003273 (+ 44us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 11:44:59.003275 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 11:44:59.003289 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 11:44:59.003657 (+ 368us) server_negotiation.cc:226] Negotiated authn=SASL
0620 11:44:59.003939 (+ 282us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 11:44:59.004796 (+ 857us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 11:44:59.005399 (+ 603us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 11:44:59.005798 (+ 399us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 11:44:59.005890 (+ 92us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 11:44:59.008150 (+ 2260us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 11:44:59.008157 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 11:44:59.008157 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 11:44:59.008162 (+ 5us) server_negotiation.cc:922] calling sasl_server_start()
0620 11:44:59.009881 (+ 1719us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 11:44:59.010408 (+ 527us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 11:44:59.010411 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 11:44:59.010411 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 11:44:59.010471 (+ 60us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 11:44:59.010652 (+ 181us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 11:44:59.010653 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 11:44:59.010653 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 11:44:59.010814 (+ 161us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 11:44:59.010894 (+ 80us) server_negotiation.cc:1035] Waiting for connection context
0620 11:44:59.011005 (+ 111us) server_negotiation.cc:299] Negotiation successful
0620 11:44:59.011091 (+ 86us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":159,"thread_start_us":92,"threads_started":1}
I20260620 11:44:59.011548 21732 tablet_service.cc:1514] Processing DeleteTablet for tablet b5b6c99c41d54a56b52e4e7bbfe1cefa with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 11:44:58 UTC) from {username='kudu', principal='kudu/127.6.234.254@KRBTEST.COM'} at 127.0.0.1:48664
I20260620 11:44:59.012071 21807 tablet_replica.cc:330] stopping tablet replica
I20260620 11:44:59.010130 21339 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin'} at 127.0.0.1:52318:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20260620 11:44:59.012195 21807 raft_consensus.cc:2229] T b5b6c99c41d54a56b52e4e7bbfe1cefa P 741432935b6d42c38ab97037b7ab9b6f [term 1 FOLLOWER]: Raft consensus shutting down.
I20260620 11:44:59.012272 21807 raft_consensus.cc:2258] T b5b6c99c41d54a56b52e4e7bbfe1cefa P 741432935b6d42c38ab97037b7ab9b6f [term 1 FOLLOWER]: Raft consensus is shut down!
W20260620 11:44:59.012307 21339 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260620 11:44:59.012595 21807 ts_tablet_manager.cc:1905] T b5b6c99c41d54a56b52e4e7bbfe1cefa P 741432935b6d42c38ab97037b7ab9b6f: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 11:44:59.014509 21732 tablet_service.cc:1467] Processing CreateTablet for tablet 72f988e4152142b8a81536ae2360ab45 (DEFAULT_TABLE table=test-table [id=abb05fd3ff1a40b0802a0a24561ea451]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 11:44:59.014644 21732 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 72f988e4152142b8a81536ae2360ab45. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 11:44:59.014714 21445 tablet_service.cc:1467] Processing CreateTablet for tablet 72f988e4152142b8a81536ae2360ab45 (DEFAULT_TABLE table=test-table [id=abb05fd3ff1a40b0802a0a24561ea451]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 11:44:59.014899 21445 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 72f988e4152142b8a81536ae2360ab45. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 11:44:59.015091 21807 ts_tablet_manager.cc:1918] T b5b6c99c41d54a56b52e4e7bbfe1cefa P 741432935b6d42c38ab97037b7ab9b6f: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260620 11:44:59.015152 21807 log.cc:1196] T b5b6c99c41d54a56b52e4e7bbfe1cefa P 741432935b6d42c38ab97037b7ab9b6f: Deleting WAL directory at /tmp/dist-test-task1JRg7N/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781955746769051-7083-0/minicluster-data/ts-2/wal/wals/b5b6c99c41d54a56b52e4e7bbfe1cefa
I20260620 11:44:59.015390 21807 ts_tablet_manager.cc:1939] T b5b6c99c41d54a56b52e4e7bbfe1cefa P 741432935b6d42c38ab97037b7ab9b6f: Deleting consensus metadata
I20260620 11:44:59.015894 21324 catalog_manager.cc:4841] TS 741432935b6d42c38ab97037b7ab9b6f (127.6.234.195:44517): tablet b5b6c99c41d54a56b52e4e7bbfe1cefa (table test-table [id=70697314689944c496e80b022368cd0d]) successfully deleted
I20260620 11:44:59.016031 21690 tablet_bootstrap.cc:492] T 72f988e4152142b8a81536ae2360ab45 P 741432935b6d42c38ab97037b7ab9b6f: Bootstrap starting.
I20260620 11:44:59.016336 21690 tablet_bootstrap.cc:654] T 72f988e4152142b8a81536ae2360ab45 P 741432935b6d42c38ab97037b7ab9b6f: Neither blocks nor log segments found. Creating new log.
I20260620 11:44:59.016345 21419 tablet_bootstrap.cc:492] T 72f988e4152142b8a81536ae2360ab45 P 6705ef55d9584b59865e2da234933258: Bootstrap starting.
I20260620 11:44:59.016629 21419 tablet_bootstrap.cc:654] T 72f988e4152142b8a81536ae2360ab45 P 6705ef55d9584b59865e2da234933258: Neither blocks nor log segments found. Creating new log.
I20260620 11:44:59.017114 21419 tablet_bootstrap.cc:492] T 72f988e4152142b8a81536ae2360ab45 P 6705ef55d9584b59865e2da234933258: No bootstrap required, opened a new log
I20260620 11:44:59.017184 21419 ts_tablet_manager.cc:1397] T 72f988e4152142b8a81536ae2360ab45 P 6705ef55d9584b59865e2da234933258: Time spent bootstrapping tablet: real 0.001s user 0.000s sys 0.000s
I20260620 11:44:59.017278 21690 tablet_bootstrap.cc:492] T 72f988e4152142b8a81536ae2360ab45 P 741432935b6d42c38ab97037b7ab9b6f: No bootstrap required, opened a new log
I20260620 11:44:59.017326 21690 ts_tablet_manager.cc:1397] T 72f988e4152142b8a81536ae2360ab45 P 741432935b6d42c38ab97037b7ab9b6f: Time spent bootstrapping tablet: real 0.001s user 0.000s sys 0.001s
I20260620 11:44:59.017298 21419 raft_consensus.cc:348] T 72f988e4152142b8a81536ae2360ab45 P 6705ef55d9584b59865e2da234933258 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fd553ed4672d4f84a1058e83d69d0e88" member_type: VOTER last_known_addr { host: "127.6.234.194" port: 40017 } } peers { permanent_uuid: "741432935b6d42c38ab97037b7ab9b6f" member_type: VOTER last_known_addr { host: "127.6.234.195" port: 44517 } } peers { permanent_uuid: "6705ef55d9584b59865e2da234933258" member_type: VOTER last_known_addr { host: "127.6.234.193" port: 35733 } }
I20260620 11:44:59.017386 21419 raft_consensus.cc:374] T 72f988e4152142b8a81536ae2360ab45 P 6705ef55d9584b59865e2da234933258 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 11:44:59.017441 21419 raft_consensus.cc:729] T 72f988e4152142b8a81536ae2360ab45 P 6705ef55d9584b59865e2da234933258 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6705ef55d9584b59865e2da234933258, State: Initialized, Role: FOLLOWER
I20260620 11:44:59.017452 21690 raft_consensus.cc:348] T 72f988e4152142b8a81536ae2360ab45 P 741432935b6d42c38ab97037b7ab9b6f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fd553ed4672d4f84a1058e83d69d0e88" member_type: VOTER last_known_addr { host: "127.6.234.194" port: 40017 } } peers { permanent_uuid: "741432935b6d42c38ab97037b7ab9b6f" member_type: VOTER last_known_addr { host: "127.6.234.195" port: 44517 } } peers { permanent_uuid: "6705ef55d9584b59865e2da234933258" member_type: VOTER last_known_addr { host: "127.6.234.193" port: 35733 } }
I20260620 11:44:59.017490 21419 consensus_queue.cc:260] T 72f988e4152142b8a81536ae2360ab45 P 6705ef55d9584b59865e2da234933258 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fd553ed4672d4f84a1058e83d69d0e88" member_type: VOTER last_known_addr { host: "127.6.234.194" port: 40017 } } peers { permanent_uuid: "741432935b6d42c38ab97037b7ab9b6f" member_type: VOTER last_known_addr { host: "127.6.234.195" port: 44517 } } peers { permanent_uuid: "6705ef55d9584b59865e2da234933258" member_type: VOTER last_known_addr { host: "127.6.234.193" port: 35733 } }
I20260620 11:44:59.017513 21690 raft_consensus.cc:374] T 72f988e4152142b8a81536ae2360ab45 P 741432935b6d42c38ab97037b7ab9b6f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 11:44:59.017549 21690 raft_consensus.cc:729] T 72f988e4152142b8a81536ae2360ab45 P 741432935b6d42c38ab97037b7ab9b6f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 741432935b6d42c38ab97037b7ab9b6f, State: Initialized, Role: FOLLOWER
I20260620 11:44:59.017594 21419 ts_tablet_manager.cc:1428] T 72f988e4152142b8a81536ae2360ab45 P 6705ef55d9584b59865e2da234933258: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
I20260620 11:44:59.017601 21690 consensus_queue.cc:260] T 72f988e4152142b8a81536ae2360ab45 P 741432935b6d42c38ab97037b7ab9b6f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fd553ed4672d4f84a1058e83d69d0e88" member_type: VOTER last_known_addr { host: "127.6.234.194" port: 40017 } } peers { permanent_uuid: "741432935b6d42c38ab97037b7ab9b6f" member_type: VOTER last_known_addr { host: "127.6.234.195" port: 44517 } } peers { permanent_uuid: "6705ef55d9584b59865e2da234933258" member_type: VOTER last_known_addr { host: "127.6.234.193" port: 35733 } }
I20260620 11:44:59.017740 21690 ts_tablet_manager.cc:1428] T 72f988e4152142b8a81536ae2360ab45 P 741432935b6d42c38ab97037b7ab9b6f: Time spent starting tablet: real 0.000s user 0.000s sys 0.001s
I20260620 11:44:59.018774 21534 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 11:44:59.014246 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.234.194:40017 (local address 127.0.0.1:57466)
0620 11:44:59.014282 (+ 36us) negotiation.cc:107] Waiting for socket to connect
0620 11:44:59.014295 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0620 11:44:59.014338 (+ 43us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 11:44:59.014748 (+ 410us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 11:44:59.014749 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 11:44:59.014754 (+ 5us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 11:44:59.014793 (+ 39us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 11:44:59.014794 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 11:44:59.016141 (+ 1347us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 11:44:59.016142 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 11:44:59.017296 (+ 1154us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 11:44:59.017300 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 11:44:59.018594 (+ 1294us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 11:44:59.018595 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 11:44:59.018642 (+ 47us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 11:44:59.018645 (+ 3us) client_negotiation.cc:769] Sending connection context
0620 11:44:59.018695 (+ 50us) client_negotiation.cc:240] Negotiation successful
0620 11:44:59.018722 (+ 27us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":19}
I20260620 11:44:59.018791 21808 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 11:44:59.014401 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57466 (local address 127.6.234.194:40017)
0620 11:44:59.014574 (+ 173us) server_negotiation.cc:206] Beginning negotiation
0620 11:44:59.014578 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 11:44:59.014589 (+ 11us) server_negotiation.cc:407] Connection header received
0620 11:44:59.014627 (+ 38us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 11:44:59.014629 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 11:44:59.014644 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 11:44:59.014863 (+ 219us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 11:44:59.014930 (+ 67us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 11:44:59.015986 (+ 1056us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 11:44:59.017877 (+ 1891us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 11:44:59.018527 (+ 650us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 11:44:59.018583 (+ 56us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 11:44:59.018619 (+ 36us) server_negotiation.cc:1035] Waiting for connection context
0620 11:44:59.018706 (+ 87us) server_negotiation.cc:299] Negotiation successful
0620 11:44:59.018731 (+ 25us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":138,"thread_start_us":105,"threads_started":1}
I20260620 11:44:59.019411 21599 tablet_service.cc:1467] Processing CreateTablet for tablet 72f988e4152142b8a81536ae2360ab45 (DEFAULT_TABLE table=test-table [id=abb05fd3ff1a40b0802a0a24561ea451]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 11:44:59.019769 21599 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 72f988e4152142b8a81536ae2360ab45. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 11:44:59.022395 21811 tablet_bootstrap.cc:492] T 72f988e4152142b8a81536ae2360ab45 P fd553ed4672d4f84a1058e83d69d0e88: Bootstrap starting.
I20260620 11:44:59.022924 21811 tablet_bootstrap.cc:654] T 72f988e4152142b8a81536ae2360ab45 P fd553ed4672d4f84a1058e83d69d0e88: Neither blocks nor log segments found. Creating new log.
I20260620 11:44:59.023391 21811 log.cc:824] T 72f988e4152142b8a81536ae2360ab45 P fd553ed4672d4f84a1058e83d69d0e88: Log is configured to *not* fsync() on all Append() calls
I20260620 11:44:59.024058 21811 tablet_bootstrap.cc:492] T 72f988e4152142b8a81536ae2360ab45 P fd553ed4672d4f84a1058e83d69d0e88: No bootstrap required, opened a new log
I20260620 11:44:59.024137 21811 ts_tablet_manager.cc:1397] T 72f988e4152142b8a81536ae2360ab45 P fd553ed4672d4f84a1058e83d69d0e88: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260620 11:44:59.025568 21811 raft_consensus.cc:348] T 72f988e4152142b8a81536ae2360ab45 P fd553ed4672d4f84a1058e83d69d0e88 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fd553ed4672d4f84a1058e83d69d0e88" member_type: VOTER last_known_addr { host: "127.6.234.194" port: 40017 } } peers { permanent_uuid: "741432935b6d42c38ab97037b7ab9b6f" member_type: VOTER last_known_addr { host: "127.6.234.195" port: 44517 } } peers { permanent_uuid: "6705ef55d9584b59865e2da234933258" member_type: VOTER last_known_addr { host: "127.6.234.193" port: 35733 } }
I20260620 11:44:59.025743 21811 raft_consensus.cc:374] T 72f988e4152142b8a81536ae2360ab45 P fd553ed4672d4f84a1058e83d69d0e88 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 11:44:59.025815 21811 raft_consensus.cc:729] T 72f988e4152142b8a81536ae2360ab45 P fd553ed4672d4f84a1058e83d69d0e88 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: fd553ed4672d4f84a1058e83d69d0e88, State: Initialized, Role: FOLLOWER
I20260620 11:44:59.025945 21811 consensus_queue.cc:260] T 72f988e4152142b8a81536ae2360ab45 P fd553ed4672d4f84a1058e83d69d0e88 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fd553ed4672d4f84a1058e83d69d0e88" member_type: VOTER last_known_addr { host: "127.6.234.194" port: 40017 } } peers { permanent_uuid: "741432935b6d42c38ab97037b7ab9b6f" member_type: VOTER last_known_addr { host: "127.6.234.195" port: 44517 } } peers { permanent_uuid: "6705ef55d9584b59865e2da234933258" member_type: VOTER last_known_addr { host: "127.6.234.193" port: 35733 } }
I20260620 11:44:59.026221 21811 ts_tablet_manager.cc:1428] T 72f988e4152142b8a81536ae2360ab45 P fd553ed4672d4f84a1058e83d69d0e88: Time spent starting tablet: real 0.002s user 0.000s sys 0.002s
I20260620 11:44:59.026238 21665 heartbeater.cc:503] Master 127.6.234.254:44633 was elected leader, sending a full tablet report...
W20260620 11:44:59.082609 21666 tablet.cc:2369] T 72f988e4152142b8a81536ae2360ab45 P fd553ed4672d4f84a1058e83d69d0e88: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260620 11:44:59.155985 21804 raft_consensus.cc:482] T 72f988e4152142b8a81536ae2360ab45 P 741432935b6d42c38ab97037b7ab9b6f [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260620 11:44:59.156177 21804 raft_consensus.cc:504] T 72f988e4152142b8a81536ae2360ab45 P 741432935b6d42c38ab97037b7ab9b6f [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fd553ed4672d4f84a1058e83d69d0e88" member_type: VOTER last_known_addr { host: "127.6.234.194" port: 40017 } } peers { permanent_uuid: "741432935b6d42c38ab97037b7ab9b6f" member_type: VOTER last_known_addr { host: "127.6.234.195" port: 44517 } } peers { permanent_uuid: "6705ef55d9584b59865e2da234933258" member_type: VOTER last_known_addr { host: "127.6.234.193" port: 35733 } }
I20260620 11:44:59.156536 21804 leader_election.cc:288] T 72f988e4152142b8a81536ae2360ab45 P 741432935b6d42c38ab97037b7ab9b6f [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers fd553ed4672d4f84a1058e83d69d0e88 (127.6.234.194:40017), 6705ef55d9584b59865e2da234933258 (127.6.234.193:35733)
I20260620 11:44:59.160585 21800 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 11:44:59.156776 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.234.193:35733 (local address 127.6.234.195:55843)
0620 11:44:59.156848 (+ 72us) negotiation.cc:107] Waiting for socket to connect
0620 11:44:59.156862 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0620 11:44:59.156917 (+ 55us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 11:44:59.157143 (+ 226us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 11:44:59.157144 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 11:44:59.157147 (+ 3us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 11:44:59.157233 (+ 86us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 11:44:59.157235 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 11:44:59.158189 (+ 954us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 11:44:59.158190 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 11:44:59.159523 (+ 1333us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 11:44:59.159526 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 11:44:59.160296 (+ 770us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 11:44:59.160297 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 11:44:59.160371 (+ 74us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 11:44:59.160397 (+ 26us) client_negotiation.cc:769] Sending connection context
0620 11:44:59.160459 (+ 62us) client_negotiation.cc:240] Negotiation successful
0620 11:44:59.160514 (+ 55us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":52,"mutex_wait_us":92}
I20260620 11:44:59.160601 21808 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 11:44:59.156855 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.6.234.195:34351 (local address 127.6.234.194:40017)
0620 11:44:59.156902 (+ 47us) server_negotiation.cc:206] Beginning negotiation
0620 11:44:59.156904 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 11:44:59.157038 (+ 134us) server_negotiation.cc:407] Connection header received
0620 11:44:59.157086 (+ 48us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 11:44:59.157087 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 11:44:59.157097 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 11:44:59.157147 (+ 50us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 11:44:59.157315 (+ 168us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 11:44:59.158100 (+ 785us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 11:44:59.159616 (+ 1516us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 11:44:59.160178 (+ 562us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 11:44:59.160252 (+ 74us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 11:44:59.160260 (+ 8us) server_negotiation.cc:1035] Waiting for connection context
0620 11:44:59.160505 (+ 245us) server_negotiation.cc:299] Negotiation successful
0620 11:44:59.160545 (+ 40us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":23}
I20260620 11:44:59.160600 21535 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 11:44:59.156863 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.6.234.195:55843 (local address 127.6.234.193:35733)
0620 11:44:59.156904 (+ 41us) server_negotiation.cc:206] Beginning negotiation
0620 11:44:59.156906 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 11:44:59.156916 (+ 10us) server_negotiation.cc:407] Connection header received
0620 11:44:59.156997 (+ 81us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 11:44:59.156998 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 11:44:59.157005 (+ 7us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 11:44:59.157101 (+ 96us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 11:44:59.157315 (+ 214us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 11:44:59.158110 (+ 795us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 11:44:59.159616 (+ 1506us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 11:44:59.160187 (+ 571us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 11:44:59.160252 (+ 65us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 11:44:59.160260 (+ 8us) server_negotiation.cc:1035] Waiting for connection context
0620 11:44:59.160468 (+ 208us) server_negotiation.cc:299] Negotiation successful
0620 11:44:59.160521 (+ 53us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":20}
I20260620 11:44:59.160826 21814 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 11:44:59.156778 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.234.194:40017 (local address 127.6.234.195:34351)
0620 11:44:59.156963 (+ 185us) negotiation.cc:107] Waiting for socket to connect
0620 11:44:59.156972 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0620 11:44:59.157025 (+ 53us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 11:44:59.157165 (+ 140us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 11:44:59.157165 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 11:44:59.157167 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 11:44:59.157233 (+ 66us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 11:44:59.157235 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 11:44:59.158189 (+ 954us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 11:44:59.158190 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 11:44:59.159523 (+ 1333us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 11:44:59.159526 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 11:44:59.160310 (+ 784us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 11:44:59.160310 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 11:44:59.160416 (+ 106us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 11:44:59.160419 (+ 3us) client_negotiation.cc:769] Sending connection context
0620 11:44:59.160712 (+ 293us) client_negotiation.cc:240] Negotiation successful
0620 11:44:59.160754 (+ 42us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":169,"mutex_wait_us":98,"thread_start_us":101,"threads_started":1}
I20260620 11:44:59.161199 21619 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "72f988e4152142b8a81536ae2360ab45" candidate_uuid: "741432935b6d42c38ab97037b7ab9b6f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "fd553ed4672d4f84a1058e83d69d0e88" is_pre_election: true
I20260620 11:44:59.161212 21481 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "72f988e4152142b8a81536ae2360ab45" candidate_uuid: "741432935b6d42c38ab97037b7ab9b6f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6705ef55d9584b59865e2da234933258" is_pre_election: true
I20260620 11:44:59.161381 21619 raft_consensus.cc:2454] T 72f988e4152142b8a81536ae2360ab45 P fd553ed4672d4f84a1058e83d69d0e88 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 741432935b6d42c38ab97037b7ab9b6f in term 0.
I20260620 11:44:59.161381 21481 raft_consensus.cc:2454] T 72f988e4152142b8a81536ae2360ab45 P 6705ef55d9584b59865e2da234933258 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 741432935b6d42c38ab97037b7ab9b6f in term 0.
I20260620 11:44:59.161597 21685 leader_election.cc:302] T 72f988e4152142b8a81536ae2360ab45 P 741432935b6d42c38ab97037b7ab9b6f [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 741432935b6d42c38ab97037b7ab9b6f, fd553ed4672d4f84a1058e83d69d0e88; no voters:
I20260620 11:44:59.161783 21804 raft_consensus.cc:2781] T 72f988e4152142b8a81536ae2360ab45 P 741432935b6d42c38ab97037b7ab9b6f [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260620 11:44:59.161826 21804 raft_consensus.cc:482] T 72f988e4152142b8a81536ae2360ab45 P 741432935b6d42c38ab97037b7ab9b6f [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260620 11:44:59.161844 21804 raft_consensus.cc:3037] T 72f988e4152142b8a81536ae2360ab45 P 741432935b6d42c38ab97037b7ab9b6f [term 0 FOLLOWER]: Advancing to term 1
I20260620 11:44:59.162516 21804 raft_consensus.cc:504] T 72f988e4152142b8a81536ae2360ab45 P 741432935b6d42c38ab97037b7ab9b6f [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fd553ed4672d4f84a1058e83d69d0e88" member_type: VOTER last_known_addr { host: "127.6.234.194" port: 40017 } } peers { permanent_uuid: "741432935b6d42c38ab97037b7ab9b6f" member_type: VOTER last_known_addr { host: "127.6.234.195" port: 44517 } } peers { permanent_uuid: "6705ef55d9584b59865e2da234933258" member_type: VOTER last_known_addr { host: "127.6.234.193" port: 35733 } }
I20260620 11:44:59.162657 21804 leader_election.cc:288] T 72f988e4152142b8a81536ae2360ab45 P 741432935b6d42c38ab97037b7ab9b6f [CANDIDATE]: Term 1 election: Requested vote from peers fd553ed4672d4f84a1058e83d69d0e88 (127.6.234.194:40017), 6705ef55d9584b59865e2da234933258 (127.6.234.193:35733)
I20260620 11:44:59.162807 21619 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "72f988e4152142b8a81536ae2360ab45" candidate_uuid: "741432935b6d42c38ab97037b7ab9b6f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "fd553ed4672d4f84a1058e83d69d0e88"
I20260620 11:44:59.162823 21481 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "72f988e4152142b8a81536ae2360ab45" candidate_uuid: "741432935b6d42c38ab97037b7ab9b6f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6705ef55d9584b59865e2da234933258"
I20260620 11:44:59.162890 21481 raft_consensus.cc:3037] T 72f988e4152142b8a81536ae2360ab45 P 6705ef55d9584b59865e2da234933258 [term 0 FOLLOWER]: Advancing to term 1
I20260620 11:44:59.162887 21619 raft_consensus.cc:3037] T 72f988e4152142b8a81536ae2360ab45 P fd553ed4672d4f84a1058e83d69d0e88 [term 0 FOLLOWER]: Advancing to term 1
I20260620 11:44:59.163770 21619 raft_consensus.cc:2454] T 72f988e4152142b8a81536ae2360ab45 P fd553ed4672d4f84a1058e83d69d0e88 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 741432935b6d42c38ab97037b7ab9b6f in term 1.
I20260620 11:44:59.163770 21481 raft_consensus.cc:2454] T 72f988e4152142b8a81536ae2360ab45 P 6705ef55d9584b59865e2da234933258 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 741432935b6d42c38ab97037b7ab9b6f in term 1.
I20260620 11:44:59.163954 21687 leader_election.cc:302] T 72f988e4152142b8a81536ae2360ab45 P 741432935b6d42c38ab97037b7ab9b6f [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 6705ef55d9584b59865e2da234933258, 741432935b6d42c38ab97037b7ab9b6f; no voters:
I20260620 11:44:59.164050 21804 raft_consensus.cc:2781] T 72f988e4152142b8a81536ae2360ab45 P 741432935b6d42c38ab97037b7ab9b6f [term 1 FOLLOWER]: Leader election won for term 1
I20260620 11:44:59.164206 21804 raft_consensus.cc:686] T 72f988e4152142b8a81536ae2360ab45 P 741432935b6d42c38ab97037b7ab9b6f [term 1 LEADER]: Becoming Leader. State: Replica: 741432935b6d42c38ab97037b7ab9b6f, State: Running, Role: LEADER
I20260620 11:44:59.164268 21804 consensus_queue.cc:237] T 72f988e4152142b8a81536ae2360ab45 P 741432935b6d42c38ab97037b7ab9b6f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fd553ed4672d4f84a1058e83d69d0e88" member_type: VOTER last_known_addr { host: "127.6.234.194" port: 40017 } } peers { permanent_uuid: "741432935b6d42c38ab97037b7ab9b6f" member_type: VOTER last_known_addr { host: "127.6.234.195" port: 44517 } } peers { permanent_uuid: "6705ef55d9584b59865e2da234933258" member_type: VOTER last_known_addr { host: "127.6.234.193" port: 35733 } }
I20260620 11:44:59.165150 21339 catalog_manager.cc:5495] T 72f988e4152142b8a81536ae2360ab45 P 741432935b6d42c38ab97037b7ab9b6f reported cstate change: term changed from 0 to 1, leader changed from <none> to 741432935b6d42c38ab97037b7ab9b6f (127.6.234.195). New cstate: current_term: 1 leader_uuid: "741432935b6d42c38ab97037b7ab9b6f" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fd553ed4672d4f84a1058e83d69d0e88" member_type: VOTER last_known_addr { host: "127.6.234.194" port: 40017 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "741432935b6d42c38ab97037b7ab9b6f" member_type: VOTER last_known_addr { host: "127.6.234.195" port: 44517 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "6705ef55d9584b59865e2da234933258" member_type: VOTER last_known_addr { host: "127.6.234.193" port: 35733 } health_report { overall_health: UNKNOWN } } }
W20260620 11:44:59.191901 21528 tablet.cc:2369] T 72f988e4152142b8a81536ae2360ab45 P 6705ef55d9584b59865e2da234933258: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260620 11:44:59.201640 21805 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 11:44:59.199446 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48666 (local address 127.6.234.195:44517)
0620 11:44:59.199508 (+ 62us) server_negotiation.cc:206] Beginning negotiation
0620 11:44:59.199510 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 11:44:59.199518 (+ 8us) server_negotiation.cc:407] Connection header received
0620 11:44:59.199552 (+ 34us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 11:44:59.199553 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 11:44:59.199570 (+ 17us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 11:44:59.199633 (+ 63us) server_negotiation.cc:226] Negotiated authn=TOKEN
0620 11:44:59.199748 (+ 115us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 11:44:59.200237 (+ 489us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 11:44:59.200910 (+ 673us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 11:44:59.201131 (+ 221us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 11:44:59.201203 (+ 72us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 11:44:59.201328 (+ 125us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0620 11:44:59.201422 (+ 94us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0620 11:44:59.201475 (+ 53us) server_negotiation.cc:1035] Waiting for connection context
0620 11:44:59.201562 (+ 87us) server_negotiation.cc:299] Negotiation successful
0620 11:44:59.201583 (+ 21us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":32}
I20260620 11:44:59.203310 21481 raft_consensus.cc:1261] T 72f988e4152142b8a81536ae2360ab45 P 6705ef55d9584b59865e2da234933258 [term 1 FOLLOWER]: Refusing update from remote peer 741432935b6d42c38ab97037b7ab9b6f: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 11:44:59.203339 21619 raft_consensus.cc:1261] T 72f988e4152142b8a81536ae2360ab45 P fd553ed4672d4f84a1058e83d69d0e88 [term 1 FOLLOWER]: Refusing update from remote peer 741432935b6d42c38ab97037b7ab9b6f: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 11:44:59.203640 21804 consensus_queue.cc:1035] T 72f988e4152142b8a81536ae2360ab45 P 741432935b6d42c38ab97037b7ab9b6f [LEADER]: Connected to new peer: Peer: permanent_uuid: "6705ef55d9584b59865e2da234933258" member_type: VOTER last_known_addr { host: "127.6.234.193" port: 35733 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 11:44:59.203811 21815 consensus_queue.cc:1035] T 72f988e4152142b8a81536ae2360ab45 P 741432935b6d42c38ab97037b7ab9b6f [LEADER]: Connected to new peer: Peer: permanent_uuid: "fd553ed4672d4f84a1058e83d69d0e88" member_type: VOTER last_known_addr { host: "127.6.234.194" port: 40017 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 11:44:59.206000 21819 mvcc.cc:205] Tried to move back new op lower bound from 7298891363133853696 to 7298891362977513472. Current Snapshot: MvccSnapshot[applied={T|T < 7298891363133853696}]
I20260620 11:44:59.207046 21821 mvcc.cc:205] Tried to move back new op lower bound from 7298891363133853696 to 7298891362977513472. Current Snapshot: MvccSnapshot[applied={T|T < 7298891363133853696 or (T in {7298891363133853696})}]
I20260620 11:44:59.208014 21339 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin'} at 127.0.0.1:52318:
table { table_name: "test-table" } modify_external_catalogs: true
I20260620 11:44:59.208107 21339 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin'} at 127.0.0.1:52318:
table { table_name: "test-table" } modify_external_catalogs: true
I20260620 11:44:59.208808 21339 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 4195d863c1404c0a8137a88b6d8ad6f1: Sending DeleteTablet for 3 replicas of tablet 72f988e4152142b8a81536ae2360ab45
I20260620 11:44:59.209110 21599 tablet_service.cc:1514] Processing DeleteTablet for tablet 72f988e4152142b8a81536ae2360ab45 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 11:44:59 UTC) from {username='kudu', principal='kudu/127.6.234.254@KRBTEST.COM'} at 127.0.0.1:57466
I20260620 11:44:59.209177 21732 tablet_service.cc:1514] Processing DeleteTablet for tablet 72f988e4152142b8a81536ae2360ab45 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 11:44:59 UTC) from {username='kudu', principal='kudu/127.6.234.254@KRBTEST.COM'} at 127.0.0.1:48664
I20260620 11:44:59.209306 21445 tablet_service.cc:1514] Processing DeleteTablet for tablet 72f988e4152142b8a81536ae2360ab45 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 11:44:59 UTC) from {username='kudu', principal='kudu/127.6.234.254@KRBTEST.COM'} at 127.0.0.1:52752
I20260620 11:44:59.209343 21827 tablet_replica.cc:330] T 72f988e4152142b8a81536ae2360ab45 P fd553ed4672d4f84a1058e83d69d0e88: stopping tablet replica
I20260620 11:44:59.209396 21807 tablet_replica.cc:330] T 72f988e4152142b8a81536ae2360ab45 P 741432935b6d42c38ab97037b7ab9b6f: stopping tablet replica
I20260620 11:44:59.209406 21536 tablet_replica.cc:330] T 72f988e4152142b8a81536ae2360ab45 P 6705ef55d9584b59865e2da234933258: stopping tablet replica
I20260620 11:44:59.209430 21827 raft_consensus.cc:2229] T 72f988e4152142b8a81536ae2360ab45 P fd553ed4672d4f84a1058e83d69d0e88 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260620 11:44:59.209443 21807 raft_consensus.cc:2229] T 72f988e4152142b8a81536ae2360ab45 P 741432935b6d42c38ab97037b7ab9b6f [term 1 LEADER]: Raft consensus shutting down.
I20260620 11:44:59.209467 21536 raft_consensus.cc:2229] T 72f988e4152142b8a81536ae2360ab45 P 6705ef55d9584b59865e2da234933258 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260620 11:44:59.209509 21536 raft_consensus.cc:2258] T 72f988e4152142b8a81536ae2360ab45 P 6705ef55d9584b59865e2da234933258 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 11:44:59.209513 21807 raft_consensus.cc:2258] T 72f988e4152142b8a81536ae2360ab45 P 741432935b6d42c38ab97037b7ab9b6f [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 11:44:59.209523 21827 raft_consensus.cc:2258] T 72f988e4152142b8a81536ae2360ab45 P fd553ed4672d4f84a1058e83d69d0e88 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 11:44:59.209827 7083 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task1JRg7N/build/release/bin/kudu with pid 21377
I20260620 11:44:59.209915 21807 ts_tablet_manager.cc:1905] T 72f988e4152142b8a81536ae2360ab45 P 741432935b6d42c38ab97037b7ab9b6f: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 11:44:59.209970 21827 ts_tablet_manager.cc:1905] T 72f988e4152142b8a81536ae2360ab45 P fd553ed4672d4f84a1058e83d69d0e88: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 11:44:59.211582 21807 ts_tablet_manager.cc:1918] T 72f988e4152142b8a81536ae2360ab45 P 741432935b6d42c38ab97037b7ab9b6f: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260620 11:44:59.211584 21827 ts_tablet_manager.cc:1918] T 72f988e4152142b8a81536ae2360ab45 P fd553ed4672d4f84a1058e83d69d0e88: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260620 11:44:59.211647 21807 log.cc:1196] T 72f988e4152142b8a81536ae2360ab45 P 741432935b6d42c38ab97037b7ab9b6f: Deleting WAL directory at /tmp/dist-test-task1JRg7N/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781955746769051-7083-0/minicluster-data/ts-2/wal/wals/72f988e4152142b8a81536ae2360ab45
I20260620 11:44:59.211655 21827 log.cc:1196] T 72f988e4152142b8a81536ae2360ab45 P fd553ed4672d4f84a1058e83d69d0e88: Deleting WAL directory at /tmp/dist-test-task1JRg7N/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781955746769051-7083-0/minicluster-data/ts-1/wal/wals/72f988e4152142b8a81536ae2360ab45
I20260620 11:44:59.211886 21807 ts_tablet_manager.cc:1939] T 72f988e4152142b8a81536ae2360ab45 P 741432935b6d42c38ab97037b7ab9b6f: Deleting consensus metadata
I20260620 11:44:59.211922 21827 ts_tablet_manager.cc:1939] T 72f988e4152142b8a81536ae2360ab45 P fd553ed4672d4f84a1058e83d69d0e88: Deleting consensus metadata
I20260620 11:44:59.212235 21324 catalog_manager.cc:4841] TS 741432935b6d42c38ab97037b7ab9b6f (127.6.234.195:44517): tablet 72f988e4152142b8a81536ae2360ab45 (table test-table [id=abb05fd3ff1a40b0802a0a24561ea451]) successfully deleted
I20260620 11:44:59.212317 21324 catalog_manager.cc:4841] TS fd553ed4672d4f84a1058e83d69d0e88 (127.6.234.194:40017): tablet 72f988e4152142b8a81536ae2360ab45 (table test-table [id=abb05fd3ff1a40b0802a0a24561ea451]) successfully deleted
W20260620 11:44:59.215155 21326 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.6.234.193:35733 (error 108)
I20260620 11:44:59.215440 7083 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task1JRg7N/build/release/bin/kudu with pid 21532
I20260620 11:44:59.216017 21534 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 11:44:59.215522 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.234.193:35733 (local address 127.0.0.1:52768)
0620 11:44:59.215942 (+ 420us) negotiation.cc:107] Waiting for socket to connect
0620 11:44:59.215985 (+ 43us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.234.193:35733: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":400}
W20260620 11:44:59.216853 21326 catalog_manager.cc:4584] TS 6705ef55d9584b59865e2da234933258 (127.6.234.193:35733): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 72f988e4152142b8a81536ae2360ab45: Network error: Client connection negotiation failed: client connection to 127.6.234.193:35733: connect: Connection refused (error 111)
I20260620 11:44:59.220180 7083 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task1JRg7N/build/release/bin/kudu with pid 21668
I20260620 11:44:59.224843 7083 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task1JRg7N/build/release/bin/kudu with pid 21306
I20260620 11:44:59.230052 7083 external_mini_cluster.cc:1326] Running /tmp/dist-test-task1JRg7N/build/release/bin/kudu
/tmp/dist-test-task1JRg7N/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task1JRg7N/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781955746769051-7083-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task1JRg7N/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781955746769051-7083-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task1JRg7N/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781955746769051-7083-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task1JRg7N/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781955746769051-7083-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.6.234.254:44633
--webserver_interface=127.6.234.254
--webserver_port=39859
--keytab_file=/tmp/dist-test-task1JRg7N/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781955746769051-7083-0/krb5kdc/kudu_127.6.234.254.keytab
--principal=kudu/127.6.234.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.6.234.212:34463
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.6.234.254:44633
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task1JRg7N/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781955746769051-7083-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task1JRg7N/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781955746769051-7083-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task1JRg7N/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781955746769051-7083-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 11:44:59.311125 21829 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 11:44:59.311321 21829 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 11:44:59.311342 21829 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 11:44:59.312474 21829 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 11:44:59.312529 21829 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 11:44:59.312556 21829 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 11:44:59.312575 21829 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 11:44:59.312595 21829 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 11:44:59.314638 21829 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.6.234.212:34463
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task1JRg7N/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781955746769051-7083-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task1JRg7N/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781955746769051-7083-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.6.234.254:44633
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task1JRg7N/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781955746769051-7083-0/krb5kdc/kudu_127.6.234.254.keytab
--principal=kudu/127.6.234.254
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.6.234.254:44633
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task1JRg7N/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781955746769051-7083-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.6.234.254
--webserver_port=39859
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.21829
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task1JRg7N/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781955746769051-7083-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 20 Jun 2026 10:32:37 UTC on ccycloud.quasar-bgyofv.root.comops.site
build id 4457173
I20260620 11:44:59.314918 21829 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 11:44:59.315145 21829 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 11:44:59.319888 21837 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 11:44:59.319916 21834 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 11:44:59.320047 21829 server_base.cc:1034] running on GCE node
W20260620 11:44:59.319890 21835 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 11:44:59.320340 21829 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 11:44:59.320574 21829 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 11:44:59.321746 21829 hybrid_clock.cc:648] HybridClock initialized: now 1781955899321734 us; error 27 us; skew 500 ppm
Jun 20 11:44:59 dist-test-slave-cjc6 krb5kdc[20762](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781955899, etypes {rep=17 tkt=17 ses=17}, kudu/127.6.234.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 11:44:59.324383 21829 init.cc:377] Logged in from keytab as kudu/127.6.234.254@KRBTEST.COM (short username kudu)
I20260620 11:44:59.324859 21829 webserver.cc:460] Webserver started at http://127.6.234.254:39859/ using document root <none> and password file <none>
I20260620 11:44:59.325102 21829 fs_manager.cc:359] Metadata directory not provided
I20260620 11:44:59.325148 21829 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 11:44:59.326514 21829 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.000s sys 0.001s
I20260620 11:44:59.327171 21844 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 11:44:59.327360 21829 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.001s
I20260620 11:44:59.327445 21829 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task1JRg7N/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781955746769051-7083-0/minicluster-data/master-0/data,/tmp/dist-test-task1JRg7N/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781955746769051-7083-0/minicluster-data/master-0/wal
uuid: "4195d863c1404c0a8137a88b6d8ad6f1"
format_stamp: "Formatted at 2026-06-20 11:44:57 on dist-test-slave-cjc6"
server_key: "6bc565ce9f35dadb384c5433eecd9a63"
server_key_iv: "9b7b41592b4da7fed09776909d1dedc9"
server_key_version: "encryptionkey@0"
I20260620 11:44:59.327780 21829 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task1JRg7N/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781955746769051-7083-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task1JRg7N/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781955746769051-7083-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task1JRg7N/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781955746769051-7083-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 11:44:59.337663 21829 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 11:44:59.339941 21829 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 11:44:59.340051 21829 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 11:44:59.344094 21829 rpc_server.cc:307] RPC server started. Bound to: 127.6.234.254:44633
I20260620 11:44:59.344125 21896 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.6.234.254:44633 every 8 connection(s)
I20260620 11:44:59.344573 21829 server_base.cc:1166] Dumped server information to /tmp/dist-test-task1JRg7N/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781955746769051-7083-0/minicluster-data/master-0/data/info.pb
I20260620 11:44:59.346160 21897 sys_catalog.cc:263] Verifying existing consensus state
I20260620 11:44:59.346802 21897 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 4195d863c1404c0a8137a88b6d8ad6f1: Bootstrap starting.
I20260620 11:44:59.354717 7083 external_mini_cluster.cc:1388] Started /tmp/dist-test-task1JRg7N/build/release/bin/kudu as pid 21829
I20260620 11:44:59.355019 7083 external_mini_cluster.cc:1326] Running /tmp/dist-test-task1JRg7N/build/release/bin/kudu
/tmp/dist-test-task1JRg7N/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task1JRg7N/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781955746769051-7083-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-task1JRg7N/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781955746769051-7083-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task1JRg7N/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781955746769051-7083-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-task1JRg7N/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781955746769051-7083-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.6.234.193:35733
--local_ip_for_outbound_sockets=127.6.234.193
--tserver_master_addrs=127.6.234.254:44633
--webserver_port=45589
--webserver_interface=127.6.234.193
--keytab_file=/tmp/dist-test-task1JRg7N/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781955746769051-7083-0/krb5kdc/kudu_127.6.234.193.keytab
--principal=kudu/127.6.234.193
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.6.234.212:34463
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task1JRg7N/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781955746769051-7083-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task1JRg7N/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781955746769051-7083-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task1JRg7N/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781955746769051-7083-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260620 11:44:59.355012 21897 log.cc:824] T 00000000000000000000000000000000 P 4195d863c1404c0a8137a88b6d8ad6f1: Log is configured to *not* fsync() on all Append() calls
I20260620 11:44:59.357113 21897 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 4195d863c1404c0a8137a88b6d8ad6f1: Bootstrap replayed 1/1 log segments. Stats: ops{read=14 overwritten=0 applied=14 ignored=0} inserts{seen=7 ignored=0} mutations{seen=9 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260620 11:44:59.357412 21897 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 4195d863c1404c0a8137a88b6d8ad6f1: Bootstrap complete.
I20260620 11:44:59.359664 21897 raft_consensus.cc:348] T 00000000000000000000000000000000 P 4195d863c1404c0a8137a88b6d8ad6f1 [term 2 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4195d863c1404c0a8137a88b6d8ad6f1" member_type: VOTER last_known_addr { host: "127.6.234.254" port: 44633 } }
I20260620 11:44:59.359903 21897 raft_consensus.cc:729] T 00000000000000000000000000000000 P 4195d863c1404c0a8137a88b6d8ad6f1 [term 2 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4195d863c1404c0a8137a88b6d8ad6f1, State: Initialized, Role: FOLLOWER
I20260620 11:44:59.360082 21897 consensus_queue.cc:260] T 00000000000000000000000000000000 P 4195d863c1404c0a8137a88b6d8ad6f1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 14, Last appended: 2.14, Last appended by leader: 14, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4195d863c1404c0a8137a88b6d8ad6f1" member_type: VOTER last_known_addr { host: "127.6.234.254" port: 44633 } }
I20260620 11:44:59.360162 21897 raft_consensus.cc:388] T 00000000000000000000000000000000 P 4195d863c1404c0a8137a88b6d8ad6f1 [term 2 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260620 11:44:59.360211 21897 raft_consensus.cc:482] T 00000000000000000000000000000000 P 4195d863c1404c0a8137a88b6d8ad6f1 [term 2 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260620 11:44:59.360258 21897 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 4195d863c1404c0a8137a88b6d8ad6f1 [term 2 FOLLOWER]: Advancing to term 3
I20260620 11:44:59.361670 21897 raft_consensus.cc:504] T 00000000000000000000000000000000 P 4195d863c1404c0a8137a88b6d8ad6f1 [term 3 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4195d863c1404c0a8137a88b6d8ad6f1" member_type: VOTER last_known_addr { host: "127.6.234.254" port: 44633 } }
I20260620 11:44:59.361819 21897 leader_election.cc:302] T 00000000000000000000000000000000 P 4195d863c1404c0a8137a88b6d8ad6f1 [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 4195d863c1404c0a8137a88b6d8ad6f1; no voters:
I20260620 11:44:59.361972 21897 leader_election.cc:288] T 00000000000000000000000000000000 P 4195d863c1404c0a8137a88b6d8ad6f1 [CANDIDATE]: Term 3 election: Requested vote from peers
I20260620 11:44:59.362005 21901 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 4195d863c1404c0a8137a88b6d8ad6f1 [term 3 FOLLOWER]: Leader election won for term 3
I20260620 11:44:59.362118 21901 raft_consensus.cc:686] T 00000000000000000000000000000000 P 4195d863c1404c0a8137a88b6d8ad6f1 [term 3 LEADER]: Becoming Leader. State: Replica: 4195d863c1404c0a8137a88b6d8ad6f1, State: Running, Role: LEADER
I20260620 11:44:59.362226 21897 sys_catalog.cc:564] T 00000000000000000000000000000000 P 4195d863c1404c0a8137a88b6d8ad6f1 [sys.catalog]: configured and running, proceeding with master startup.
I20260620 11:44:59.362203 21901 consensus_queue.cc:237] T 00000000000000000000000000000000 P 4195d863c1404c0a8137a88b6d8ad6f1 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 14, Committed index: 14, Last appended: 2.14, Last appended by leader: 14, Current term: 3, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4195d863c1404c0a8137a88b6d8ad6f1" member_type: VOTER last_known_addr { host: "127.6.234.254" port: 44633 } }
I20260620 11:44:59.362473 21901 sys_catalog.cc:455] T 00000000000000000000000000000000 P 4195d863c1404c0a8137a88b6d8ad6f1 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 3 leader_uuid: "4195d863c1404c0a8137a88b6d8ad6f1" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4195d863c1404c0a8137a88b6d8ad6f1" member_type: VOTER last_known_addr { host: "127.6.234.254" port: 44633 } } }
I20260620 11:44:59.362543 21901 sys_catalog.cc:458] T 00000000000000000000000000000000 P 4195d863c1404c0a8137a88b6d8ad6f1 [sys.catalog]: This master's current role is: LEADER
I20260620 11:44:59.362660 21901 sys_catalog.cc:455] T 00000000000000000000000000000000 P 4195d863c1404c0a8137a88b6d8ad6f1 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 4195d863c1404c0a8137a88b6d8ad6f1. Latest consensus state: current_term: 3 leader_uuid: "4195d863c1404c0a8137a88b6d8ad6f1" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4195d863c1404c0a8137a88b6d8ad6f1" member_type: VOTER last_known_addr { host: "127.6.234.254" port: 44633 } } }
I20260620 11:44:59.362730 21901 sys_catalog.cc:458] T 00000000000000000000000000000000 P 4195d863c1404c0a8137a88b6d8ad6f1 [sys.catalog]: This master's current role is: LEADER
I20260620 11:44:59.363051 21905 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260620 11:44:59.363921 21905 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260620 11:44:59.364042 21905 catalog_manager.cc:1259] Loaded cluster ID: 19042a500bc94fdabc05bb1fbd473859
I20260620 11:44:59.364265 21905 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
F20260620 11:44:59.364493 21905 catalog_manager.cc:1466] Initializing Kudu internal certificate authority failed: Runtime error: unable to load data from memory: error:0D0680A8:asn1 encoding routines:ASN1_CHECK_TLEN:wrong tag:tasn_dec.c:1220 error:0D06C03A:asn1 encoding routines:ASN1_D2I_EX_PRIMITIVE:nested asn1 error:tasn_dec.c:788 error:0D08303A:asn1 encoding routines:ASN1_TEMPLATE_NOEXP_D2I:nested asn1 error:tasn_dec.c:720:Field=version, Type=RSA error:04093004:rsa routines:OLD_RSA_PRIV_DECODE:RSA lib:rsa_ameth.c:121 error:0D0680A8:asn1 encoding routines:ASN1_CHECK_TLEN:wrong tag:tasn_dec.c:1220 error:0D06C03A:asn1 encoding routines:ASN1_D2I_EX_PRIMITIVE:nested asn1 error:tasn_dec.c:788 error:0D08303A:asn1 encoding routines:ASN1_TEMPLATE_NOEXP_D2I:nested asn1 error:tasn_dec.c:720:Field=version, Type=PKCS8_PRIV_KEY_INFO
*** Check failure stack trace: ***
@ 0x7f138f3639ad google::LogMessage::Fail() at ??:0
@ 0x7f138f36688a google::LogMessage::SendToLog() at ??:0
@ 0x7f138f3634de google::LogMessage::Flush() at ??:0
@ 0x7f138f364a19 google::LogMessageFatal::~LogMessageFatal() at ??:0
@ 0x7f13972840f1 _ZZN4kudu6master14CatalogManager24PrepareForLeadershipTaskEvENKUlSt8functionIFNS_6StatusEvEERKNS_9consensus13RaftConsensusElPKcE_clES5_S9_lSB_.isra.591 at ??:0
@ 0x7f139728d8b9 kudu::master::CatalogManager::PrepareForLeadershipTask() at ??:0
@ 0x7f13912310d3 kudu::ThreadPool::DispatchThread() at ??:0
@ 0x7f1391226954 kudu::Thread::SuperviseThread() at ??:0
@ 0x7f139208a6db start_thread at ??:0
@ 0x7f138de6161f clone at ??:0
W20260620 11:44:59.447062 21900 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 11:44:59.447297 21900 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 11:44:59.447338 21900 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 11:44:59.448498 21900 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 11:44:59.448567 21900 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 11:44:59.448613 21900 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.6.234.193
I20260620 11:44:59.450232 21900 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.6.234.212:34463
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task1JRg7N/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781955746769051-7083-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-task1JRg7N/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781955746769051-7083-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task1JRg7N/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781955746769051-7083-0/krb5kdc/kudu_127.6.234.193.keytab
--principal=kudu/127.6.234.193
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.6.234.193:35733
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task1JRg7N/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781955746769051-7083-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.6.234.193
--webserver_port=45589
--tserver_master_addrs=127.6.234.254:44633
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.21900
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.6.234.193
--log_dir=/tmp/dist-test-task1JRg7N/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781955746769051-7083-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 20 Jun 2026 10:32:37 UTC on ccycloud.quasar-bgyofv.root.comops.site
build id 4457173
I20260620 11:44:59.450508 21900 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 11:44:59.450737 21900 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 11:44:59.456090 21934 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 11:44:59.456102 21933 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 11:44:59.456113 21936 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 11:44:59.456382 21900 server_base.cc:1034] running on GCE node
I20260620 11:44:59.456593 21900 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 11:44:59.456844 21900 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 11:44:59.458005 21900 hybrid_clock.cc:648] HybridClock initialized: now 1781955899457991 us; error 35 us; skew 500 ppm
Jun 20 11:44:59 dist-test-slave-cjc6 krb5kdc[20762](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781955899, etypes {rep=17 tkt=17 ses=17}, kudu/127.6.234.193@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 11:44:59.460585 21900 init.cc:377] Logged in from keytab as kudu/127.6.234.193@KRBTEST.COM (short username kudu)
I20260620 11:44:59.461066 21900 webserver.cc:460] Webserver started at http://127.6.234.193:45589/ using document root <none> and password file <none>
I20260620 11:44:59.461310 21900 fs_manager.cc:359] Metadata directory not provided
I20260620 11:44:59.461390 21900 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 11:44:59.462888 21900 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.001s sys 0.000s
I20260620 11:44:59.463670 21943 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 11:44:59.463855 21900 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260620 11:44:59.463955 21900 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task1JRg7N/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781955746769051-7083-0/minicluster-data/ts-0/data,/tmp/dist-test-task1JRg7N/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781955746769051-7083-0/minicluster-data/ts-0/wal
uuid: "6705ef55d9584b59865e2da234933258"
format_stamp: "Formatted at 2026-06-20 11:44:57 on dist-test-slave-cjc6"
server_key: "aaecbf620d1e90b359cbec9b916a3ad9"
server_key_iv: "36d40e1318d339ab95f1f04a89b23817"
server_key_version: "encryptionkey@0"
I20260620 11:44:59.464339 21900 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task1JRg7N/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781955746769051-7083-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-task1JRg7N/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781955746769051-7083-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-task1JRg7N/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781955746769051-7083-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 11:44:59.476956 21900 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 11:44:59.479351 21900 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 11:44:59.479522 21900 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 11:44:59.479755 21900 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 11:44:59.480227 21950 ts_tablet_manager.cc:536] Loading tablet metadata (0/1 complete)
I20260620 11:44:59.482354 21900 ts_tablet_manager.cc:579] Loaded tablet metadata (1 total tablets, 1 live tablets)
I20260620 11:44:59.482420 21900 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.002s user 0.000s sys 0.000s
I20260620 11:44:59.482465 21900 ts_tablet_manager.cc:594] Registering tablets (0/1 complete)
I20260620 11:44:59.483104 21900 ts_tablet_manager.cc:610] Registered 1 tablets
I20260620 11:44:59.483160 21900 ts_tablet_manager.cc:589] Time spent register tablets: real 0.001s user 0.000s sys 0.000s
I20260620 11:44:59.483228 21950 tablet_bootstrap.cc:492] T 72f988e4152142b8a81536ae2360ab45 P 6705ef55d9584b59865e2da234933258: Bootstrap starting.
I20260620 11:44:59.490610 21900 rpc_server.cc:307] RPC server started. Bound to: 127.6.234.193:35733
I20260620 11:44:59.491000 21900 server_base.cc:1166] Dumped server information to /tmp/dist-test-task1JRg7N/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781955746769051-7083-0/minicluster-data/ts-0/data/info.pb
I20260620 11:44:59.491544 22057 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.6.234.193:35733 every 8 connection(s)
I20260620 11:44:59.494522 21950 log.cc:824] T 72f988e4152142b8a81536ae2360ab45 P 6705ef55d9584b59865e2da234933258: Log is configured to *not* fsync() on all Append() calls
I20260620 11:44:59.497423 21950 tablet_bootstrap.cc:492] T 72f988e4152142b8a81536ae2360ab45 P 6705ef55d9584b59865e2da234933258: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=1 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260620 11:44:59.497802 21950 tablet_bootstrap.cc:492] T 72f988e4152142b8a81536ae2360ab45 P 6705ef55d9584b59865e2da234933258: Bootstrap complete.
I20260620 11:44:59.499068 21950 ts_tablet_manager.cc:1397] T 72f988e4152142b8a81536ae2360ab45 P 6705ef55d9584b59865e2da234933258: Time spent bootstrapping tablet: real 0.016s user 0.005s sys 0.010s
I20260620 11:44:59.500268 7083 external_mini_cluster.cc:1388] Started /tmp/dist-test-task1JRg7N/build/release/bin/kudu as pid 21900
I20260620 11:44:59.500563 7083 external_mini_cluster.cc:1326] Running /tmp/dist-test-task1JRg7N/build/release/bin/kudu
/tmp/dist-test-task1JRg7N/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task1JRg7N/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781955746769051-7083-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-task1JRg7N/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781955746769051-7083-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task1JRg7N/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781955746769051-7083-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-task1JRg7N/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781955746769051-7083-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.6.234.194:40017
--local_ip_for_outbound_sockets=127.6.234.194
--tserver_master_addrs=127.6.234.254:44633
--webserver_port=40019
--webserver_interface=127.6.234.194
--keytab_file=/tmp/dist-test-task1JRg7N/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781955746769051-7083-0/krb5kdc/kudu_127.6.234.194.keytab
--principal=kudu/127.6.234.194
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.6.234.212:34463
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task1JRg7N/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781955746769051-7083-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task1JRg7N/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781955746769051-7083-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task1JRg7N/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781955746769051-7083-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260620 11:44:59.500694 21950 raft_consensus.cc:348] T 72f988e4152142b8a81536ae2360ab45 P 6705ef55d9584b59865e2da234933258 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fd553ed4672d4f84a1058e83d69d0e88" member_type: VOTER last_known_addr { host: "127.6.234.194" port: 40017 } } peers { permanent_uuid: "741432935b6d42c38ab97037b7ab9b6f" member_type: VOTER last_known_addr { host: "127.6.234.195" port: 44517 } } peers { permanent_uuid: "6705ef55d9584b59865e2da234933258" member_type: VOTER last_known_addr { host: "127.6.234.193" port: 35733 } }
I20260620 11:44:59.501021 21950 raft_consensus.cc:729] T 72f988e4152142b8a81536ae2360ab45 P 6705ef55d9584b59865e2da234933258 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6705ef55d9584b59865e2da234933258, State: Initialized, Role: FOLLOWER
I20260620 11:44:59.501163 21950 consensus_queue.cc:260] T 72f988e4152142b8a81536ae2360ab45 P 6705ef55d9584b59865e2da234933258 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fd553ed4672d4f84a1058e83d69d0e88" member_type: VOTER last_known_addr { host: "127.6.234.194" port: 40017 } } peers { permanent_uuid: "741432935b6d42c38ab97037b7ab9b6f" member_type: VOTER last_known_addr { host: "127.6.234.195" port: 44517 } } peers { permanent_uuid: "6705ef55d9584b59865e2da234933258" member_type: VOTER last_known_addr { host: "127.6.234.193" port: 35733 } }
I20260620 11:44:59.501425 21950 ts_tablet_manager.cc:1428] T 72f988e4152142b8a81536ae2360ab45 P 6705ef55d9584b59865e2da234933258: Time spent starting tablet: real 0.002s user 0.001s sys 0.002s
W20260620 11:44:59.578863 22060 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 11:44:59.491711 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.234.254:44633 (local address 127.6.234.193:53671)
0620 11:44:59.491899 (+ 188us) negotiation.cc:107] Waiting for socket to connect
0620 11:44:59.491942 (+ 43us) client_negotiation.cc:174] Beginning negotiation
0620 11:44:59.492427 (+ 485us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 11:44:59.578645 (+ 86218us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.234.254:44633: BlockingRecv error: recv error from unknown peer: Transport endpoint is not connected (error 107)
Metrics: {"client-negotiator.queue_time_us":141,"thread_start_us":118,"threads_started":1}
W20260620 11:44:59.579149 22058 heartbeater.cc:644] Failed to heartbeat to 127.6.234.254:44633 (0 consecutive failures): Network error: Failed to ping master at 127.6.234.254:44633: Client connection negotiation failed: client connection to 127.6.234.254:44633: BlockingRecv error: recv error from unknown peer: Transport endpoint is not connected (error 107)
I20260620 11:44:59.579545 22060 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 11:44:59.579407 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.234.254:44633 (local address 127.6.234.193:47505)
0620 11:44:59.579469 (+ 62us) negotiation.cc:107] Waiting for socket to connect
0620 11:44:59.579518 (+ 49us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.234.254:44633: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":40}
I20260620 11:44:59.579932 22060 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 11:44:59.579835 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.234.254:44633 (local address 127.6.234.193:51035)
0620 11:44:59.579881 (+ 46us) negotiation.cc:107] Waiting for socket to connect
0620 11:44:59.579910 (+ 29us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.234.254:44633: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":31}
W20260620 11:44:59.580101 22058 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
W20260620 11:44:59.582863 22063 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 11:44:59.583045 22063 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 11:44:59.583068 22063 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 11:44:59.584267 22063 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 11:44:59.584306 22063 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 11:44:59.584337 22063 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.6.234.194
I20260620 11:44:59.585875 22063 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.6.234.212:34463
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task1JRg7N/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781955746769051-7083-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-task1JRg7N/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781955746769051-7083-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task1JRg7N/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781955746769051-7083-0/krb5kdc/kudu_127.6.234.194.keytab
--principal=kudu/127.6.234.194
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.6.234.194:40017
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task1JRg7N/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781955746769051-7083-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.6.234.194
--webserver_port=40019
--tserver_master_addrs=127.6.234.254:44633
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.22063
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.6.234.194
--log_dir=/tmp/dist-test-task1JRg7N/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781955746769051-7083-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 20 Jun 2026 10:32:37 UTC on ccycloud.quasar-bgyofv.root.comops.site
build id 4457173
I20260620 11:44:59.586102 22063 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 11:44:59.586320 22063 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 11:44:59.591315 22069 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 11:44:59.591385 22071 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 11:44:59.591362 22068 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 11:44:59.591524 22063 server_base.cc:1034] running on GCE node
I20260620 11:44:59.591753 22063 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 11:44:59.591988 22063 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 11:44:59.593128 22063 hybrid_clock.cc:648] HybridClock initialized: now 1781955899593108 us; error 34 us; skew 500 ppm
Jun 20 11:44:59 dist-test-slave-cjc6 krb5kdc[20762](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781955899, etypes {rep=17 tkt=17 ses=17}, kudu/127.6.234.194@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 11:44:59.595588 22063 init.cc:377] Logged in from keytab as kudu/127.6.234.194@KRBTEST.COM (short username kudu)
I20260620 11:44:59.596030 22063 webserver.cc:460] Webserver started at http://127.6.234.194:40019/ using document root <none> and password file <none>
I20260620 11:44:59.596208 22063 fs_manager.cc:359] Metadata directory not provided
I20260620 11:44:59.596247 22063 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 11:44:59.597466 22063 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.000s sys 0.001s
I20260620 11:44:59.598062 22078 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 11:44:59.598344 22063 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.000s
I20260620 11:44:59.598451 22063 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task1JRg7N/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781955746769051-7083-0/minicluster-data/ts-1/data,/tmp/dist-test-task1JRg7N/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781955746769051-7083-0/minicluster-data/ts-1/wal
uuid: "fd553ed4672d4f84a1058e83d69d0e88"
format_stamp: "Formatted at 2026-06-20 11:44:57 on dist-test-slave-cjc6"
server_key: "175cb27c7fe999d5d8de3c5eb757ab3a"
server_key_iv: "7a8318dc18ca4f5755ecaf130d21ff5d"
server_key_version: "encryptionkey@0"
I20260620 11:44:59.598785 22063 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task1JRg7N/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781955746769051-7083-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-task1JRg7N/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781955746769051-7083-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-task1JRg7N/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781955746769051-7083-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 11:44:59.608245 22063 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 11:44:59.610126 22063 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 11:44:59.610282 22063 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 11:44:59.610508 22063 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 11:44:59.610803 22063 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 11:44:59.610863 22063 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 11:44:59.610903 22063 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 11:44:59.610929 22063 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 11:44:59.616959 22063 rpc_server.cc:307] RPC server started. Bound to: 127.6.234.194:40017
I20260620 11:44:59.617002 22191 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.6.234.194:40017 every 8 connection(s)
I20260620 11:44:59.617362 22063 server_base.cc:1166] Dumped server information to /tmp/dist-test-task1JRg7N/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781955746769051-7083-0/minicluster-data/ts-1/data/info.pb
I20260620 11:44:59.618428 22194 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 11:44:59.618082 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.234.254:44633 (local address 127.6.234.194:55281)
0620 11:44:59.618223 (+ 141us) negotiation.cc:107] Waiting for socket to connect
0620 11:44:59.618288 (+ 65us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.234.254:44633: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":106,"thread_start_us":65,"threads_started":1}
W20260620 11:44:59.618841 22192 heartbeater.cc:644] Failed to heartbeat to 127.6.234.254:44633 (0 consecutive failures): Network error: Failed to ping master at 127.6.234.254:44633: Client connection negotiation failed: client connection to 127.6.234.254:44633: connect: Connection refused (error 111)
I20260620 11:44:59.619103 22194 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 11:44:59.619020 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.234.254:44633 (local address 127.6.234.194:44301)
0620 11:44:59.619062 (+ 42us) negotiation.cc:107] Waiting for socket to connect
0620 11:44:59.619088 (+ 26us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.234.254:44633: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":28}
I20260620 11:44:59.619364 22194 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 11:44:59.619287 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.234.254:44633 (local address 127.6.234.194:43581)
0620 11:44:59.619329 (+ 42us) negotiation.cc:107] Waiting for socket to connect
0620 11:44:59.619350 (+ 21us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.234.254:44633: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":30}
W20260620 11:44:59.619479 22192 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
I20260620 11:44:59.626362 7083 external_mini_cluster.cc:1388] Started /tmp/dist-test-task1JRg7N/build/release/bin/kudu as pid 22063
I20260620 11:44:59.626629 7083 external_mini_cluster.cc:1326] Running /tmp/dist-test-task1JRg7N/build/release/bin/kudu
/tmp/dist-test-task1JRg7N/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task1JRg7N/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781955746769051-7083-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-task1JRg7N/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781955746769051-7083-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task1JRg7N/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781955746769051-7083-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-task1JRg7N/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781955746769051-7083-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.6.234.195:44517
--local_ip_for_outbound_sockets=127.6.234.195
--tserver_master_addrs=127.6.234.254:44633
--webserver_port=35283
--webserver_interface=127.6.234.195
--keytab_file=/tmp/dist-test-task1JRg7N/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781955746769051-7083-0/krb5kdc/kudu_127.6.234.195.keytab
--principal=kudu/127.6.234.195
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.6.234.212:34463
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task1JRg7N/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781955746769051-7083-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task1JRg7N/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781955746769051-7083-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task1JRg7N/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781955746769051-7083-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 11:44:59.706583 22195 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 11:44:59.706786 22195 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 11:44:59.706809 22195 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 11:44:59.707878 22195 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 11:44:59.707911 22195 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 11:44:59.707937 22195 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.6.234.195
I20260620 11:44:59.709272 22195 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.6.234.212:34463
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task1JRg7N/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781955746769051-7083-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-task1JRg7N/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781955746769051-7083-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task1JRg7N/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781955746769051-7083-0/krb5kdc/kudu_127.6.234.195.keytab
--principal=kudu/127.6.234.195
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.6.234.195:44517
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task1JRg7N/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781955746769051-7083-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.6.234.195
--webserver_port=35283
--tserver_master_addrs=127.6.234.254:44633
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.22195
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.6.234.195
--log_dir=/tmp/dist-test-task1JRg7N/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781955746769051-7083-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 20 Jun 2026 10:32:37 UTC on ccycloud.quasar-bgyofv.root.comops.site
build id 4457173
I20260620 11:44:59.709453 22195 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 11:44:59.709625 22195 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 11:44:59.714619 22201 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 11:44:59.714627 22203 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 11:44:59.714619 22200 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 11:44:59.714867 22195 server_base.cc:1034] running on GCE node
I20260620 11:44:59.715080 22195 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 11:44:59.715318 22195 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 11:44:59.716461 22195 hybrid_clock.cc:648] HybridClock initialized: now 1781955899716449 us; error 32 us; skew 500 ppm
Jun 20 11:44:59 dist-test-slave-cjc6 krb5kdc[20762](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781955899, etypes {rep=17 tkt=17 ses=17}, kudu/127.6.234.195@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 11:44:59.718947 22195 init.cc:377] Logged in from keytab as kudu/127.6.234.195@KRBTEST.COM (short username kudu)
I20260620 11:44:59.719321 22195 webserver.cc:460] Webserver started at http://127.6.234.195:35283/ using document root <none> and password file <none>
I20260620 11:44:59.719493 22195 fs_manager.cc:359] Metadata directory not provided
I20260620 11:44:59.719532 22195 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 11:44:59.720758 22195 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.001s sys 0.000s
I20260620 11:44:59.721347 22210 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 11:44:59.721520 22195 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260620 11:44:59.721624 22195 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task1JRg7N/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781955746769051-7083-0/minicluster-data/ts-2/data,/tmp/dist-test-task1JRg7N/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781955746769051-7083-0/minicluster-data/ts-2/wal
uuid: "741432935b6d42c38ab97037b7ab9b6f"
format_stamp: "Formatted at 2026-06-20 11:44:58 on dist-test-slave-cjc6"
server_key: "0fdbd66a6280cdebd7c4853a2e880ec0"
server_key_iv: "3a8b79d819f0fbd10c123d784ae2fdaf"
server_key_version: "encryptionkey@0"
I20260620 11:44:59.722019 22195 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task1JRg7N/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781955746769051-7083-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-task1JRg7N/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781955746769051-7083-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-task1JRg7N/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781955746769051-7083-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 11:44:59.763811 22195 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 11:44:59.765770 22195 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 11:44:59.765937 22195 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 11:44:59.766175 22195 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 11:44:59.766481 22195 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 11:44:59.766536 22195 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 11:44:59.766579 22195 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 11:44:59.766604 22195 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 11:44:59.772581 22195 rpc_server.cc:307] RPC server started. Bound to: 127.6.234.195:44517
I20260620 11:44:59.772624 22323 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.6.234.195:44517 every 8 connection(s)
I20260620 11:44:59.772959 22195 server_base.cc:1166] Dumped server information to /tmp/dist-test-task1JRg7N/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781955746769051-7083-0/minicluster-data/ts-2/data/info.pb
I20260620 11:44:59.774006 22326 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 11:44:59.773603 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.234.254:44633 (local address 127.6.234.195:59979)
0620 11:44:59.773794 (+ 191us) negotiation.cc:107] Waiting for socket to connect
0620 11:44:59.773869 (+ 75us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.234.254:44633: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":154,"thread_start_us":91,"threads_started":1}
W20260620 11:44:59.774386 22324 heartbeater.cc:644] Failed to heartbeat to 127.6.234.254:44633 (0 consecutive failures): Network error: Failed to ping master at 127.6.234.254:44633: Client connection negotiation failed: client connection to 127.6.234.254:44633: connect: Connection refused (error 111)
I20260620 11:44:59.774616 22326 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 11:44:59.774550 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.234.254:44633 (local address 127.6.234.195:50687)
0620 11:44:59.774576 (+ 26us) negotiation.cc:107] Waiting for socket to connect
0620 11:44:59.774601 (+ 25us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.234.254:44633: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":15}
I20260620 11:44:59.774852 22326 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 11:44:59.774787 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.234.254:44633 (local address 127.6.234.195:56563)
0620 11:44:59.774818 (+ 31us) negotiation.cc:107] Waiting for socket to connect
0620 11:44:59.774840 (+ 22us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.234.254:44633: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":22}
W20260620 11:44:59.774953 22324 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
I20260620 11:44:59.781949 7083 external_mini_cluster.cc:1388] Started /tmp/dist-test-task1JRg7N/build/release/bin/kudu as pid 22195
I20260620 11:45:00.580708 22328 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 11:45:00.580464 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.234.254:44633 (local address 127.6.234.193:44763)
0620 11:45:00.580600 (+ 136us) negotiation.cc:107] Waiting for socket to connect
0620 11:45:00.580641 (+ 41us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.234.254:44633: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":116,"thread_start_us":86,"threads_started":1}
I20260620 11:45:00.620030 22329 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 11:45:00.619819 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.234.254:44633 (local address 127.6.234.194:52779)
0620 11:45:00.619948 (+ 129us) negotiation.cc:107] Waiting for socket to connect
0620 11:45:00.619983 (+ 35us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.234.254:44633: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":114,"thread_start_us":77,"threads_started":1}
I20260620 11:45:00.775530 22330 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 11:45:00.775297 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.234.254:44633 (local address 127.6.234.195:55523)
0620 11:45:00.775433 (+ 136us) negotiation.cc:107] Waiting for socket to connect
0620 11:45:00.775475 (+ 42us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.234.254:44633: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":116,"thread_start_us":57,"threads_started":1}
I20260620 11:45:01.044230 22331 raft_consensus.cc:482] T 72f988e4152142b8a81536ae2360ab45 P 6705ef55d9584b59865e2da234933258 [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260620 11:45:01.044358 22331 raft_consensus.cc:504] T 72f988e4152142b8a81536ae2360ab45 P 6705ef55d9584b59865e2da234933258 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fd553ed4672d4f84a1058e83d69d0e88" member_type: VOTER last_known_addr { host: "127.6.234.194" port: 40017 } } peers { permanent_uuid: "741432935b6d42c38ab97037b7ab9b6f" member_type: VOTER last_known_addr { host: "127.6.234.195" port: 44517 } } peers { permanent_uuid: "6705ef55d9584b59865e2da234933258" member_type: VOTER last_known_addr { host: "127.6.234.193" port: 35733 } }
I20260620 11:45:01.044775 22331 leader_election.cc:288] T 72f988e4152142b8a81536ae2360ab45 P 6705ef55d9584b59865e2da234933258 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers fd553ed4672d4f84a1058e83d69d0e88 (127.6.234.194:40017), 741432935b6d42c38ab97037b7ab9b6f (127.6.234.195:44517)
Jun 20 11:45:01 dist-test-slave-cjc6 krb5kdc[20762](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781955899, etypes {rep=17 tkt=17 ses=17}, kudu/127.6.234.193@KRBTEST.COM for kudu/127.6.234.195@KRBTEST.COM
Jun 20 11:45:01 dist-test-slave-cjc6 krb5kdc[20762](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781955899, etypes {rep=17 tkt=17 ses=17}, kudu/127.6.234.193@KRBTEST.COM for kudu/127.6.234.194@KRBTEST.COM
I20260620 11:45:01.055832 22332 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 11:45:01.045059 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.6.234.193:60443 (local address 127.6.234.194:40017)
0620 11:45:01.045230 (+ 171us) server_negotiation.cc:206] Beginning negotiation
0620 11:45:01.045260 (+ 30us) server_negotiation.cc:399] Waiting for connection header
0620 11:45:01.045295 (+ 35us) server_negotiation.cc:407] Connection header received
0620 11:45:01.045816 (+ 521us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 11:45:01.045819 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 11:45:01.045863 (+ 44us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 11:45:01.046018 (+ 155us) server_negotiation.cc:226] Negotiated authn=SASL
0620 11:45:01.046788 (+ 770us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 11:45:01.047622 (+ 834us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 11:45:01.048617 (+ 995us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 11:45:01.049010 (+ 393us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 11:45:01.049131 (+ 121us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 11:45:01.052450 (+ 3319us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 11:45:01.052454 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 11:45:01.052454 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 11:45:01.052463 (+ 9us) server_negotiation.cc:922] calling sasl_server_start()
0620 11:45:01.054338 (+ 1875us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 11:45:01.054674 (+ 336us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 11:45:01.054678 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 11:45:01.054678 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 11:45:01.054767 (+ 89us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 11:45:01.054982 (+ 215us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 11:45:01.054983 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 11:45:01.054983 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 11:45:01.055217 (+ 234us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 11:45:01.055303 (+ 86us) server_negotiation.cc:1035] Waiting for connection context
0620 11:45:01.055577 (+ 274us) server_negotiation.cc:299] Negotiation successful
0620 11:45:01.055715 (+ 138us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":136,"thread_start_us":59,"threads_started":1}
I20260620 11:45:01.055821 22328 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 11:45:01.044953 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.234.194:40017 (local address 127.6.234.193:60443)
0620 11:45:01.045053 (+ 100us) negotiation.cc:107] Waiting for socket to connect
0620 11:45:01.045068 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0620 11:45:01.045144 (+ 76us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 11:45:01.046047 (+ 903us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 11:45:01.046050 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 11:45:01.046447 (+ 397us) client_negotiation.cc:189] Negotiated authn=SASL
0620 11:45:01.046648 (+ 201us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 11:45:01.046654 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 11:45:01.047743 (+ 1089us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 11:45:01.047744 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 11:45:01.048523 (+ 779us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 11:45:01.048525 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 11:45:01.049102 (+ 577us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 11:45:01.049103 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 11:45:01.049224 (+ 121us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 11:45:01.049875 (+ 651us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 11:45:01.049883 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0620 11:45:01.052340 (+ 2457us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 11:45:01.054419 (+ 2079us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 11:45:01.054420 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 11:45:01.054421 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0620 11:45:01.054593 (+ 172us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 11:45:01.054846 (+ 253us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 11:45:01.054847 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 11:45:01.054847 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0620 11:45:01.054910 (+ 63us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 11:45:01.055299 (+ 389us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 11:45:01.055300 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 11:45:01.055416 (+ 116us) client_negotiation.cc:769] Sending connection context
0620 11:45:01.055518 (+ 102us) client_negotiation.cc:240] Negotiation successful
0620 11:45:01.055716 (+ 198us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":75,"mutex_wait_us":17}
I20260620 11:45:01.055868 22334 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 11:45:01.045202 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.6.234.193:53449 (local address 127.6.234.195:44517)
0620 11:45:01.045339 (+ 137us) server_negotiation.cc:206] Beginning negotiation
0620 11:45:01.045349 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0620 11:45:01.045382 (+ 33us) server_negotiation.cc:407] Connection header received
0620 11:45:01.045861 (+ 479us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 11:45:01.045864 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 11:45:01.045902 (+ 38us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 11:45:01.046047 (+ 145us) server_negotiation.cc:226] Negotiated authn=SASL
0620 11:45:01.046780 (+ 733us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 11:45:01.047643 (+ 863us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 11:45:01.048627 (+ 984us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 11:45:01.049020 (+ 393us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 11:45:01.049133 (+ 113us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 11:45:01.052044 (+ 2911us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 11:45:01.052047 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 11:45:01.052048 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 11:45:01.052057 (+ 9us) server_negotiation.cc:922] calling sasl_server_start()
0620 11:45:01.054071 (+ 2014us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 11:45:01.054562 (+ 491us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 11:45:01.054565 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 11:45:01.054566 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 11:45:01.054671 (+ 105us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 11:45:01.054965 (+ 294us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 11:45:01.054966 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 11:45:01.054966 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 11:45:01.055212 (+ 246us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 11:45:01.055299 (+ 87us) server_negotiation.cc:1035] Waiting for connection context
0620 11:45:01.055584 (+ 285us) server_negotiation.cc:299] Negotiation successful
0620 11:45:01.055747 (+ 163us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":98,"thread_start_us":75,"threads_started":1}
I20260620 11:45:01.056013 22333 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 11:45:01.045110 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.234.195:44517 (local address 127.6.234.193:53449)
0620 11:45:01.045224 (+ 114us) negotiation.cc:107] Waiting for socket to connect
0620 11:45:01.045233 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0620 11:45:01.045269 (+ 36us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 11:45:01.046047 (+ 778us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 11:45:01.046050 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 11:45:01.046449 (+ 399us) client_negotiation.cc:189] Negotiated authn=SASL
0620 11:45:01.046650 (+ 201us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 11:45:01.046655 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 11:45:01.047748 (+ 1093us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 11:45:01.047749 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 11:45:01.048523 (+ 774us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 11:45:01.048526 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 11:45:01.049115 (+ 589us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 11:45:01.049116 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 11:45:01.049222 (+ 106us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 11:45:01.049968 (+ 746us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 11:45:01.049973 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0620 11:45:01.051919 (+ 1946us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 11:45:01.054151 (+ 2232us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 11:45:01.054156 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 11:45:01.054158 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0620 11:45:01.054400 (+ 242us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 11:45:01.054755 (+ 355us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 11:45:01.054756 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 11:45:01.054757 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0620 11:45:01.054885 (+ 128us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 11:45:01.055291 (+ 406us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 11:45:01.055298 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 11:45:01.055416 (+ 118us) client_negotiation.cc:769] Sending connection context
0620 11:45:01.055517 (+ 101us) client_negotiation.cc:240] Negotiation successful
0620 11:45:01.055800 (+ 283us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":104,"mutex_wait_us":204,"thread_start_us":65,"threads_started":1}
W20260620 11:45:01.056887 21945 leader_election.cc:341] T 72f988e4152142b8a81536ae2360ab45 P 6705ef55d9584b59865e2da234933258 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 741432935b6d42c38ab97037b7ab9b6f (127.6.234.195:44517): Not found: Tablet not found: 72f988e4152142b8a81536ae2360ab45
W20260620 11:45:01.057268 21945 leader_election.cc:341] T 72f988e4152142b8a81536ae2360ab45 P 6705ef55d9584b59865e2da234933258 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer fd553ed4672d4f84a1058e83d69d0e88 (127.6.234.194:40017): Not found: Tablet not found: 72f988e4152142b8a81536ae2360ab45
I20260620 11:45:01.057312 21945 leader_election.cc:302] T 72f988e4152142b8a81536ae2360ab45 P 6705ef55d9584b59865e2da234933258 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 6705ef55d9584b59865e2da234933258; no voters: 741432935b6d42c38ab97037b7ab9b6f, fd553ed4672d4f84a1058e83d69d0e88
I20260620 11:45:01.057420 22331 raft_consensus.cc:2726] T 72f988e4152142b8a81536ae2360ab45 P 6705ef55d9584b59865e2da234933258 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260620 11:45:01.581462 22335 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 11:45:01.581245 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.234.254:44633 (local address 127.6.234.193:52145)
0620 11:45:01.581383 (+ 138us) negotiation.cc:107] Waiting for socket to connect
0620 11:45:01.581415 (+ 32us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.234.254:44633: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":121,"thread_start_us":64,"threads_started":1}
I20260620 11:45:01.620718 22336 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 11:45:01.620519 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.234.254:44633 (local address 127.6.234.194:33795)
0620 11:45:01.620640 (+ 121us) negotiation.cc:107] Waiting for socket to connect
0620 11:45:01.620680 (+ 40us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.234.254:44633: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":103,"thread_start_us":61,"threads_started":1}
I20260620 11:45:01.776368 22337 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 11:45:01.776138 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.234.254:44633 (local address 127.6.234.195:42681)
0620 11:45:01.776274 (+ 136us) negotiation.cc:107] Waiting for socket to connect
0620 11:45:01.776316 (+ 42us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.234.254:44633: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":118,"thread_start_us":72,"threads_started":1}
I20260620 11:45:02.582275 22338 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 11:45:02.582040 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.234.254:44633 (local address 127.6.234.193:38181)
0620 11:45:02.582179 (+ 139us) negotiation.cc:107] Waiting for socket to connect
0620 11:45:02.582226 (+ 47us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.234.254:44633: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":117,"thread_start_us":64,"threads_started":1}
I20260620 11:45:02.621534 22339 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 11:45:02.621258 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.234.254:44633 (local address 127.6.234.194:33961)
0620 11:45:02.621421 (+ 163us) negotiation.cc:107] Waiting for socket to connect
0620 11:45:02.621467 (+ 46us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.234.254:44633: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":121,"thread_start_us":91,"threads_started":1}
I20260620 11:45:02.777005 22340 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 11:45:02.776820 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.234.254:44633 (local address 127.6.234.195:57191)
0620 11:45:02.776934 (+ 114us) negotiation.cc:107] Waiting for socket to connect
0620 11:45:02.776967 (+ 33us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.234.254:44633: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":100,"thread_start_us":61,"threads_started":1}
I20260620 11:45:03.116973 22341 raft_consensus.cc:482] T 72f988e4152142b8a81536ae2360ab45 P 6705ef55d9584b59865e2da234933258 [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260620 11:45:03.117066 22341 raft_consensus.cc:504] T 72f988e4152142b8a81536ae2360ab45 P 6705ef55d9584b59865e2da234933258 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fd553ed4672d4f84a1058e83d69d0e88" member_type: VOTER last_known_addr { host: "127.6.234.194" port: 40017 } } peers { permanent_uuid: "741432935b6d42c38ab97037b7ab9b6f" member_type: VOTER last_known_addr { host: "127.6.234.195" port: 44517 } } peers { permanent_uuid: "6705ef55d9584b59865e2da234933258" member_type: VOTER last_known_addr { host: "127.6.234.193" port: 35733 } }
I20260620 11:45:03.117244 22341 leader_election.cc:288] T 72f988e4152142b8a81536ae2360ab45 P 6705ef55d9584b59865e2da234933258 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers fd553ed4672d4f84a1058e83d69d0e88 (127.6.234.194:40017), 741432935b6d42c38ab97037b7ab9b6f (127.6.234.195:44517)
W20260620 11:45:03.117782 21945 leader_election.cc:341] T 72f988e4152142b8a81536ae2360ab45 P 6705ef55d9584b59865e2da234933258 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 741432935b6d42c38ab97037b7ab9b6f (127.6.234.195:44517): Not found: Tablet not found: 72f988e4152142b8a81536ae2360ab45
W20260620 11:45:03.117852 21945 leader_election.cc:341] T 72f988e4152142b8a81536ae2360ab45 P 6705ef55d9584b59865e2da234933258 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer fd553ed4672d4f84a1058e83d69d0e88 (127.6.234.194:40017): Not found: Tablet not found: 72f988e4152142b8a81536ae2360ab45
I20260620 11:45:03.117868 21945 leader_election.cc:302] T 72f988e4152142b8a81536ae2360ab45 P 6705ef55d9584b59865e2da234933258 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 6705ef55d9584b59865e2da234933258; no voters: 741432935b6d42c38ab97037b7ab9b6f, fd553ed4672d4f84a1058e83d69d0e88
I20260620 11:45:03.117934 22341 raft_consensus.cc:2726] T 72f988e4152142b8a81536ae2360ab45 P 6705ef55d9584b59865e2da234933258 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260620 11:45:03.583063 22342 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 11:45:03.582837 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.234.254:44633 (local address 127.6.234.193:55291)
0620 11:45:03.582969 (+ 132us) negotiation.cc:107] Waiting for socket to connect
0620 11:45:03.583014 (+ 45us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.234.254:44633: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":110,"thread_start_us":66,"threads_started":1}
I20260620 11:45:03.622213 22343 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 11:45:03.622025 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.234.254:44633 (local address 127.6.234.194:37523)
0620 11:45:03.622135 (+ 110us) negotiation.cc:107] Waiting for socket to connect
0620 11:45:03.622176 (+ 41us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.234.254:44633: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":94,"thread_start_us":54,"threads_started":1}
I20260620 11:45:03.777679 22344 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 11:45:03.777454 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.234.254:44633 (local address 127.6.234.195:38253)
0620 11:45:03.777597 (+ 143us) negotiation.cc:107] Waiting for socket to connect
0620 11:45:03.777635 (+ 38us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.234.254:44633: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":119,"thread_start_us":58,"threads_started":1}
I20260620 11:45:04.583899 22345 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 11:45:04.583619 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.234.254:44633 (local address 127.6.234.193:43753)
0620 11:45:04.583816 (+ 197us) negotiation.cc:107] Waiting for socket to connect
0620 11:45:04.583853 (+ 37us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.234.254:44633: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":179,"thread_start_us":117,"threads_started":1}
I20260620 11:45:04.622931 22346 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 11:45:04.622682 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.234.254:44633 (local address 127.6.234.194:56619)
0620 11:45:04.622843 (+ 161us) negotiation.cc:107] Waiting for socket to connect
0620 11:45:04.622887 (+ 44us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.234.254:44633: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":141,"thread_start_us":79,"threads_started":1}
I20260620 11:45:04.778405 22347 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 11:45:04.778193 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.234.254:44633 (local address 127.6.234.195:50299)
0620 11:45:04.778330 (+ 137us) negotiation.cc:107] Waiting for socket to connect
0620 11:45:04.778366 (+ 36us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.234.254:44633: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":120,"thread_start_us":87,"threads_started":1}
I20260620 11:45:05.584753 22348 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 11:45:05.584517 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.234.254:44633 (local address 127.6.234.193:49599)
0620 11:45:05.584664 (+ 147us) negotiation.cc:107] Waiting for socket to connect
0620 11:45:05.584704 (+ 40us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.234.254:44633: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":128,"thread_start_us":80,"threads_started":1}
I20260620 11:45:05.623741 22349 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 11:45:05.623469 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.234.254:44633 (local address 127.6.234.194:41933)
0620 11:45:05.623628 (+ 159us) negotiation.cc:107] Waiting for socket to connect
0620 11:45:05.623697 (+ 69us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.234.254:44633: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":137,"thread_start_us":76,"threads_started":1}
I20260620 11:45:05.779078 22350 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 11:45:05.778866 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.234.254:44633 (local address 127.6.234.195:48199)
0620 11:45:05.778997 (+ 131us) negotiation.cc:107] Waiting for socket to connect
0620 11:45:05.779035 (+ 38us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.234.254:44633: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":112,"thread_start_us":58,"threads_started":1}
I20260620 11:45:06.252772 22351 raft_consensus.cc:482] T 72f988e4152142b8a81536ae2360ab45 P 6705ef55d9584b59865e2da234933258 [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260620 11:45:06.252889 22351 raft_consensus.cc:504] T 72f988e4152142b8a81536ae2360ab45 P 6705ef55d9584b59865e2da234933258 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fd553ed4672d4f84a1058e83d69d0e88" member_type: VOTER last_known_addr { host: "127.6.234.194" port: 40017 } } peers { permanent_uuid: "741432935b6d42c38ab97037b7ab9b6f" member_type: VOTER last_known_addr { host: "127.6.234.195" port: 44517 } } peers { permanent_uuid: "6705ef55d9584b59865e2da234933258" member_type: VOTER last_known_addr { host: "127.6.234.193" port: 35733 } }
I20260620 11:45:06.253095 22351 leader_election.cc:288] T 72f988e4152142b8a81536ae2360ab45 P 6705ef55d9584b59865e2da234933258 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers fd553ed4672d4f84a1058e83d69d0e88 (127.6.234.194:40017), 741432935b6d42c38ab97037b7ab9b6f (127.6.234.195:44517)
W20260620 11:45:06.253443 21945 leader_election.cc:341] T 72f988e4152142b8a81536ae2360ab45 P 6705ef55d9584b59865e2da234933258 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer fd553ed4672d4f84a1058e83d69d0e88 (127.6.234.194:40017): Not found: Tablet not found: 72f988e4152142b8a81536ae2360ab45
W20260620 11:45:06.253517 21945 leader_election.cc:341] T 72f988e4152142b8a81536ae2360ab45 P 6705ef55d9584b59865e2da234933258 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 741432935b6d42c38ab97037b7ab9b6f (127.6.234.195:44517): Not found: Tablet not found: 72f988e4152142b8a81536ae2360ab45
I20260620 11:45:06.253625 21945 leader_election.cc:302] T 72f988e4152142b8a81536ae2360ab45 P 6705ef55d9584b59865e2da234933258 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 6705ef55d9584b59865e2da234933258; no voters: 741432935b6d42c38ab97037b7ab9b6f, fd553ed4672d4f84a1058e83d69d0e88
I20260620 11:45:06.253717 22351 raft_consensus.cc:2726] T 72f988e4152142b8a81536ae2360ab45 P 6705ef55d9584b59865e2da234933258 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260620 11:45:06.585503 22352 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 11:45:06.585275 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.234.254:44633 (local address 127.6.234.193:37567)
0620 11:45:06.585413 (+ 138us) negotiation.cc:107] Waiting for socket to connect
0620 11:45:06.585456 (+ 43us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.234.254:44633: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":123,"thread_start_us":78,"threads_started":1}
I20260620 11:45:06.624468 22353 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 11:45:06.624240 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.234.254:44633 (local address 127.6.234.194:59781)
0620 11:45:06.624392 (+ 152us) negotiation.cc:107] Waiting for socket to connect
0620 11:45:06.624426 (+ 34us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.234.254:44633: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":135,"thread_start_us":90,"threads_started":1}
I20260620 11:45:06.779755 22354 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 11:45:06.779528 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.234.254:44633 (local address 127.6.234.195:34859)
0620 11:45:06.779673 (+ 145us) negotiation.cc:107] Waiting for socket to connect
0620 11:45:06.779711 (+ 38us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.234.254:44633: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":124,"thread_start_us":66,"threads_started":1}
I20260620 11:45:07.586241 22355 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 11:45:07.586016 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.234.254:44633 (local address 127.6.234.193:41001)
0620 11:45:07.586153 (+ 137us) negotiation.cc:107] Waiting for socket to connect
0620 11:45:07.586196 (+ 43us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.234.254:44633: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":119,"thread_start_us":74,"threads_started":1}
I20260620 11:45:07.625221 22356 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 11:45:07.624985 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.234.254:44633 (local address 127.6.234.194:33557)
0620 11:45:07.625139 (+ 154us) negotiation.cc:107] Waiting for socket to connect
0620 11:45:07.625179 (+ 40us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.234.254:44633: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":136,"thread_start_us":82,"threads_started":1}
I20260620 11:45:07.780531 22357 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 11:45:07.780288 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.234.254:44633 (local address 127.6.234.195:51529)
0620 11:45:07.780443 (+ 155us) negotiation.cc:107] Waiting for socket to connect
0620 11:45:07.780485 (+ 42us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.234.254:44633: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":137,"thread_start_us":67,"threads_started":1}
I20260620 11:45:08.586949 22358 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 11:45:08.586725 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.234.254:44633 (local address 127.6.234.193:35403)
0620 11:45:08.586854 (+ 129us) negotiation.cc:107] Waiting for socket to connect
0620 11:45:08.586902 (+ 48us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.234.254:44633: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":110,"thread_start_us":82,"threads_started":1}
I20260620 11:45:08.626088 22359 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 11:45:08.625845 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.234.254:44633 (local address 127.6.234.194:44717)
0620 11:45:08.625993 (+ 148us) negotiation.cc:107] Waiting for socket to connect
0620 11:45:08.626034 (+ 41us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.234.254:44633: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":130,"thread_start_us":60,"threads_started":1}
I20260620 11:45:08.769989 22360 raft_consensus.cc:482] T 72f988e4152142b8a81536ae2360ab45 P 6705ef55d9584b59865e2da234933258 [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260620 11:45:08.770088 22360 raft_consensus.cc:504] T 72f988e4152142b8a81536ae2360ab45 P 6705ef55d9584b59865e2da234933258 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fd553ed4672d4f84a1058e83d69d0e88" member_type: VOTER last_known_addr { host: "127.6.234.194" port: 40017 } } peers { permanent_uuid: "741432935b6d42c38ab97037b7ab9b6f" member_type: VOTER last_known_addr { host: "127.6.234.195" port: 44517 } } peers { permanent_uuid: "6705ef55d9584b59865e2da234933258" member_type: VOTER last_known_addr { host: "127.6.234.193" port: 35733 } }
I20260620 11:45:08.770255 22360 leader_election.cc:288] T 72f988e4152142b8a81536ae2360ab45 P 6705ef55d9584b59865e2da234933258 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers fd553ed4672d4f84a1058e83d69d0e88 (127.6.234.194:40017), 741432935b6d42c38ab97037b7ab9b6f (127.6.234.195:44517)
W20260620 11:45:08.770694 21945 leader_election.cc:341] T 72f988e4152142b8a81536ae2360ab45 P 6705ef55d9584b59865e2da234933258 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer fd553ed4672d4f84a1058e83d69d0e88 (127.6.234.194:40017): Not found: Tablet not found: 72f988e4152142b8a81536ae2360ab45
W20260620 11:45:08.770761 21945 leader_election.cc:341] T 72f988e4152142b8a81536ae2360ab45 P 6705ef55d9584b59865e2da234933258 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 741432935b6d42c38ab97037b7ab9b6f (127.6.234.195:44517): Not found: Tablet not found: 72f988e4152142b8a81536ae2360ab45
I20260620 11:45:08.770776 21945 leader_election.cc:302] T 72f988e4152142b8a81536ae2360ab45 P 6705ef55d9584b59865e2da234933258 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 6705ef55d9584b59865e2da234933258; no voters: 741432935b6d42c38ab97037b7ab9b6f, fd553ed4672d4f84a1058e83d69d0e88
I20260620 11:45:08.770848 22360 raft_consensus.cc:2726] T 72f988e4152142b8a81536ae2360ab45 P 6705ef55d9584b59865e2da234933258 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260620 11:45:08.781307 22361 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 11:45:08.781059 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.234.254:44633 (local address 127.6.234.195:50703)
0620 11:45:08.781208 (+ 149us) negotiation.cc:107] Waiting for socket to connect
0620 11:45:08.781248 (+ 40us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.234.254:44633: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":131,"thread_start_us":61,"threads_started":1}
I20260620 11:45:09.587628 22362 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 11:45:09.587385 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.234.254:44633 (local address 127.6.234.193:39621)
0620 11:45:09.587532 (+ 147us) negotiation.cc:107] Waiting for socket to connect
0620 11:45:09.587570 (+ 38us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.234.254:44633: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":129,"thread_start_us":78,"threads_started":1}
I20260620 11:45:09.626842 22363 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 11:45:09.626619 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.234.254:44633 (local address 127.6.234.194:42055)
0620 11:45:09.626754 (+ 135us) negotiation.cc:107] Waiting for socket to connect
0620 11:45:09.626795 (+ 41us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.234.254:44633: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":118,"thread_start_us":71,"threads_started":1}
I20260620 11:45:09.782087 22364 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 11:45:09.781881 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.234.254:44633 (local address 127.6.234.195:44111)
0620 11:45:09.781999 (+ 118us) negotiation.cc:107] Waiting for socket to connect
0620 11:45:09.782042 (+ 43us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.234.254:44633: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":102,"thread_start_us":59,"threads_started":1}
I20260620 11:45:10.588372 22365 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 11:45:10.588126 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.234.254:44633 (local address 127.6.234.193:40587)
0620 11:45:10.588278 (+ 152us) negotiation.cc:107] Waiting for socket to connect
0620 11:45:10.588319 (+ 41us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.234.254:44633: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":134,"thread_start_us":77,"threads_started":1}
I20260620 11:45:10.627545 22366 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 11:45:10.627314 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.234.254:44633 (local address 127.6.234.194:41399)
0620 11:45:10.627463 (+ 149us) negotiation.cc:107] Waiting for socket to connect
0620 11:45:10.627500 (+ 37us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.234.254:44633: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":130,"thread_start_us":65,"threads_started":1}
I20260620 11:45:10.782851 22367 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 11:45:10.782629 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.234.254:44633 (local address 127.6.234.195:49913)
0620 11:45:10.782760 (+ 131us) negotiation.cc:107] Waiting for socket to connect
0620 11:45:10.782804 (+ 44us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.234.254:44633: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":112,"thread_start_us":66,"threads_started":1}
I20260620 11:45:11.589188 22368 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 11:45:11.588950 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.234.254:44633 (local address 127.6.234.193:39733)
0620 11:45:11.589098 (+ 148us) negotiation.cc:107] Waiting for socket to connect
0620 11:45:11.589140 (+ 42us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.234.254:44633: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":126,"thread_start_us":76,"threads_started":1}
I20260620 11:45:11.628240 22369 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 11:45:11.628027 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.234.254:44633 (local address 127.6.234.194:37811)
0620 11:45:11.628161 (+ 134us) negotiation.cc:107] Waiting for socket to connect
0620 11:45:11.628195 (+ 34us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.234.254:44633: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":118,"thread_start_us":75,"threads_started":1}
I20260620 11:45:11.783587 22370 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 11:45:11.783355 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.234.254:44633 (local address 127.6.234.195:53551)
0620 11:45:11.783498 (+ 143us) negotiation.cc:107] Waiting for socket to connect
0620 11:45:11.783541 (+ 43us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.234.254:44633: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":124,"thread_start_us":76,"threads_started":1}
I20260620 11:45:12.590056 22371 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 11:45:12.589792 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.234.254:44633 (local address 127.6.234.193:51609)
0620 11:45:12.589955 (+ 163us) negotiation.cc:107] Waiting for socket to connect
0620 11:45:12.589999 (+ 44us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.234.254:44633: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":144,"thread_start_us":77,"threads_started":1}
I20260620 11:45:12.628906 22372 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 11:45:12.628674 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.234.254:44633 (local address 127.6.234.194:44387)
0620 11:45:12.628825 (+ 151us) negotiation.cc:107] Waiting for socket to connect
0620 11:45:12.628862 (+ 37us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.234.254:44633: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":134,"thread_start_us":71,"threads_started":1}
I20260620 11:45:12.784238 22373 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 11:45:12.784023 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.234.254:44633 (local address 127.6.234.195:54627)
0620 11:45:12.784158 (+ 135us) negotiation.cc:107] Waiting for socket to connect
0620 11:45:12.784192 (+ 34us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.234.254:44633: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":121,"thread_start_us":73,"threads_started":1}
I20260620 11:45:13.590750 22374 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 11:45:13.590540 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.234.254:44633 (local address 127.6.234.193:34019)
0620 11:45:13.590666 (+ 126us) negotiation.cc:107] Waiting for socket to connect
0620 11:45:13.590706 (+ 40us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.234.254:44633: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":111,"thread_start_us":68,"threads_started":1}
I20260620 11:45:13.629634 22375 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 11:45:13.629379 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.234.254:44633 (local address 127.6.234.194:59855)
0620 11:45:13.629549 (+ 170us) negotiation.cc:107] Waiting for socket to connect
0620 11:45:13.629590 (+ 41us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.234.254:44633: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":144,"thread_start_us":68,"threads_started":1}
I20260620 11:45:13.784909 22376 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 11:45:13.784706 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.234.254:44633 (local address 127.6.234.195:53601)
0620 11:45:13.784823 (+ 117us) negotiation.cc:107] Waiting for socket to connect
0620 11:45:13.784862 (+ 39us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.234.254:44633: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":101,"thread_start_us":62,"threads_started":1}
I20260620 11:45:14.591568 22377 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 11:45:14.591348 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.234.254:44633 (local address 127.6.234.193:49601)
0620 11:45:14.591478 (+ 130us) negotiation.cc:107] Waiting for socket to connect
0620 11:45:14.591518 (+ 40us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.234.254:44633: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":112,"thread_start_us":63,"threads_started":1}
I20260620 11:45:14.630509 22378 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 11:45:14.630267 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.234.254:44633 (local address 127.6.234.194:52379)
0620 11:45:14.630418 (+ 151us) negotiation.cc:107] Waiting for socket to connect
0620 11:45:14.630462 (+ 44us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.234.254:44633: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":135,"thread_start_us":74,"threads_started":1}
I20260620 11:45:14.784323 7083 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task1JRg7N/build/release/bin/kudu with pid 21900
I20260620 11:45:14.785774 22379 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 11:45:14.785433 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.234.254:44633 (local address 127.6.234.195:36171)
0620 11:45:14.785657 (+ 224us) negotiation.cc:107] Waiting for socket to connect
0620 11:45:14.785719 (+ 62us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.234.254:44633: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":196,"thread_start_us":54,"threads_started":1}
I20260620 11:45:14.788990 7083 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task1JRg7N/build/release/bin/kudu with pid 22063
I20260620 11:45:14.793150 7083 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task1JRg7N/build/release/bin/kudu with pid 22195
2026-06-20T11:45:14Z chronyd exiting
[ OK ] SecurityITest.TestEnableAndDisableEncryptedIPKICert (18462 ms)
[ RUN ] SecurityITest.TestEnableAndDisableEncryptedTSK
Loading random data
Initializing database '/tmp/dist-test-task1JRg7N/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781955746769051-7083-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 20 11:45:14 dist-test-slave-cjc6 krb5kdc[22382](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 20 11:45:14 dist-test-slave-cjc6 krb5kdc[22382](info): set up 2 sockets
Jun 20 11:45:14 dist-test-slave-cjc6 krb5kdc[22382](info): commencing operation
krb5kdc: starting...
W20260620 11:45:15.869890 7083 mini_kdc.cc:121] Time spent starting KDC: real 1.057s user 0.000s sys 0.006s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task1JRg7N/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781955746769051-7083-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task1JRg7N/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781955746769051-7083-0/krb5kdc/test-user.keytab.
Jun 20 11:45:15 dist-test-slave-cjc6 krb5kdc[22382](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781955915, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-20T11:45:15Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-20T11:45:15Z Disabled control of system clock
WARNING: no policy specified for kudu/127.6.234.254@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.6.234.254@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.6.234.254 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task1JRg7N/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781955746769051-7083-0/krb5kdc/kudu_127.6.234.254.keytab.
Entry for principal kudu/127.6.234.254 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task1JRg7N/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781955746769051-7083-0/krb5kdc/kudu_127.6.234.254.keytab.
I20260620 11:45:15.984959 7083 external_mini_cluster.cc:1326] Running /tmp/dist-test-task1JRg7N/build/release/bin/kudu
/tmp/dist-test-task1JRg7N/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task1JRg7N/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781955746769051-7083-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task1JRg7N/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781955746769051-7083-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task1JRg7N/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781955746769051-7083-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task1JRg7N/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781955746769051-7083-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.6.234.254:41033
--webserver_interface=127.6.234.254
--webserver_port=0
--keytab_file=/tmp/dist-test-task1JRg7N/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781955746769051-7083-0/krb5kdc/kudu_127.6.234.254.keytab
--principal=kudu/127.6.234.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.6.234.212:45931
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.6.234.254:41033
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--authn_token_validity_seconds=20
--authz_token_validity_seconds=20
--tsk_rotation_seconds=20 with env {KRB5CCNAME=/tmp/dist-test-task1JRg7N/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781955746769051-7083-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task1JRg7N/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781955746769051-7083-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task1JRg7N/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781955746769051-7083-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 11:45:16.065609 22395 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 11:45:16.065922 22395 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 11:45:16.065963 22395 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 11:45:16.067111 22395 flags.cc:425] Enabled experimental flag: --authn_token_validity_seconds=20
W20260620 11:45:16.067184 22395 flags.cc:425] Enabled experimental flag: --authz_token_validity_seconds=20
W20260620 11:45:16.067206 22395 flags.cc:425] Enabled experimental flag: --tsk_rotation_seconds=20
W20260620 11:45:16.067227 22395 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 11:45:16.067255 22395 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 11:45:16.067287 22395 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 11:45:16.067314 22395 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 11:45:16.067340 22395 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 11:45:16.068832 22395 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.6.234.212:45931
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task1JRg7N/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781955746769051-7083-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task1JRg7N/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781955746769051-7083-0/minicluster-data/master-0/wal
--authn_token_validity_seconds=20
--authz_token_validity_seconds=20
--tsk_rotation_seconds=20
--ipki_ca_key_size=768
--master_addresses=127.6.234.254:41033
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task1JRg7N/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781955746769051-7083-0/krb5kdc/kudu_127.6.234.254.keytab
--principal=kudu/127.6.234.254
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.6.234.254:41033
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task1JRg7N/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781955746769051-7083-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.6.234.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.22395
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task1JRg7N/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781955746769051-7083-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 20 Jun 2026 10:32:37 UTC on ccycloud.quasar-bgyofv.root.comops.site
build id 4457173
I20260620 11:45:16.069119 22395 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 11:45:16.069351 22395 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 11:45:16.074257 22403 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 11:45:16.074294 22401 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 11:45:16.074311 22395 server_base.cc:1034] running on GCE node
W20260620 11:45:16.074280 22400 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 11:45:16.074721 22395 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 11:45:16.075023 22395 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 11:45:16.076203 22395 hybrid_clock.cc:648] HybridClock initialized: now 1781955916076175 us; error 60 us; skew 500 ppm
Jun 20 11:45:16 dist-test-slave-cjc6 krb5kdc[22382](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781955916, etypes {rep=17 tkt=17 ses=17}, kudu/127.6.234.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 11:45:16.083411 22395 init.cc:377] Logged in from keytab as kudu/127.6.234.254@KRBTEST.COM (short username kudu)
I20260620 11:45:16.083942 22395 webserver.cc:460] Webserver started at http://127.6.234.254:42389/ using document root <none> and password file <none>
I20260620 11:45:16.084188 22395 fs_manager.cc:359] Metadata directory not provided
I20260620 11:45:16.084259 22395 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 11:45:16.084410 22395 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 11:45:16.085479 22395 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task1JRg7N/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781955746769051-7083-0/minicluster-data/master-0/data/instance:
uuid: "d3dfdbcc949c49839481eebfb80e05d7"
format_stamp: "Formatted at 2026-06-20 11:45:16 on dist-test-slave-cjc6"
server_key: "d0eb388e2b925cb0e115a1ea9364db71"
server_key_iv: "03dce555a5f1af3eb04507723e819dc5"
server_key_version: "encryptionkey@0"
I20260620 11:45:16.085873 22395 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task1JRg7N/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781955746769051-7083-0/minicluster-data/master-0/wal/instance:
uuid: "d3dfdbcc949c49839481eebfb80e05d7"
format_stamp: "Formatted at 2026-06-20 11:45:16 on dist-test-slave-cjc6"
server_key: "d0eb388e2b925cb0e115a1ea9364db71"
server_key_iv: "03dce555a5f1af3eb04507723e819dc5"
server_key_version: "encryptionkey@0"
I20260620 11:45:16.087214 22395 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.001s
I20260620 11:45:16.087961 22410 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 11:45:16.088131 22395 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.001s
I20260620 11:45:16.088230 22395 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task1JRg7N/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781955746769051-7083-0/minicluster-data/master-0/data,/tmp/dist-test-task1JRg7N/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781955746769051-7083-0/minicluster-data/master-0/wal
uuid: "d3dfdbcc949c49839481eebfb80e05d7"
format_stamp: "Formatted at 2026-06-20 11:45:16 on dist-test-slave-cjc6"
server_key: "d0eb388e2b925cb0e115a1ea9364db71"
server_key_iv: "03dce555a5f1af3eb04507723e819dc5"
server_key_version: "encryptionkey@0"
I20260620 11:45:16.088313 22395 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task1JRg7N/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781955746769051-7083-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task1JRg7N/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781955746769051-7083-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task1JRg7N/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781955746769051-7083-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 11:45:16.103192 22395 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 11:45:16.105195 22395 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 11:45:16.105373 22395 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 11:45:16.109512 22395 rpc_server.cc:307] RPC server started. Bound to: 127.6.234.254:41033
I20260620 11:45:16.109573 22462 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.6.234.254:41033 every 8 connection(s)
I20260620 11:45:16.109958 22395 server_base.cc:1166] Dumped server information to /tmp/dist-test-task1JRg7N/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781955746769051-7083-0/minicluster-data/master-0/data/info.pb
I20260620 11:45:16.110797 22463 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 11:45:16.112991 22463 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P d3dfdbcc949c49839481eebfb80e05d7: Bootstrap starting.
I20260620 11:45:16.113525 22463 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P d3dfdbcc949c49839481eebfb80e05d7: Neither blocks nor log segments found. Creating new log.
I20260620 11:45:16.113792 22463 log.cc:824] T 00000000000000000000000000000000 P d3dfdbcc949c49839481eebfb80e05d7: Log is configured to *not* fsync() on all Append() calls
I20260620 11:45:16.114415 22463 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P d3dfdbcc949c49839481eebfb80e05d7: No bootstrap required, opened a new log
I20260620 11:45:16.115911 22463 raft_consensus.cc:348] T 00000000000000000000000000000000 P d3dfdbcc949c49839481eebfb80e05d7 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d3dfdbcc949c49839481eebfb80e05d7" member_type: VOTER last_known_addr { host: "127.6.234.254" port: 41033 } }
I20260620 11:45:16.116048 22463 raft_consensus.cc:374] T 00000000000000000000000000000000 P d3dfdbcc949c49839481eebfb80e05d7 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 11:45:16.116071 22463 raft_consensus.cc:729] T 00000000000000000000000000000000 P d3dfdbcc949c49839481eebfb80e05d7 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d3dfdbcc949c49839481eebfb80e05d7, State: Initialized, Role: FOLLOWER
I20260620 11:45:16.116223 22463 consensus_queue.cc:260] T 00000000000000000000000000000000 P d3dfdbcc949c49839481eebfb80e05d7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d3dfdbcc949c49839481eebfb80e05d7" member_type: VOTER last_known_addr { host: "127.6.234.254" port: 41033 } }
I20260620 11:45:16.116308 22463 raft_consensus.cc:388] T 00000000000000000000000000000000 P d3dfdbcc949c49839481eebfb80e05d7 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260620 11:45:16.116331 22463 raft_consensus.cc:482] T 00000000000000000000000000000000 P d3dfdbcc949c49839481eebfb80e05d7 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260620 11:45:16.116365 22463 raft_consensus.cc:3037] T 00000000000000000000000000000000 P d3dfdbcc949c49839481eebfb80e05d7 [term 0 FOLLOWER]: Advancing to term 1
I20260620 11:45:16.116838 22463 raft_consensus.cc:504] T 00000000000000000000000000000000 P d3dfdbcc949c49839481eebfb80e05d7 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d3dfdbcc949c49839481eebfb80e05d7" member_type: VOTER last_known_addr { host: "127.6.234.254" port: 41033 } }
I20260620 11:45:16.116922 22463 leader_election.cc:302] T 00000000000000000000000000000000 P d3dfdbcc949c49839481eebfb80e05d7 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: d3dfdbcc949c49839481eebfb80e05d7; no voters:
I20260620 11:45:16.117048 22463 leader_election.cc:288] T 00000000000000000000000000000000 P d3dfdbcc949c49839481eebfb80e05d7 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260620 11:45:16.117121 22466 raft_consensus.cc:2781] T 00000000000000000000000000000000 P d3dfdbcc949c49839481eebfb80e05d7 [term 1 FOLLOWER]: Leader election won for term 1
I20260620 11:45:16.117288 22466 raft_consensus.cc:686] T 00000000000000000000000000000000 P d3dfdbcc949c49839481eebfb80e05d7 [term 1 LEADER]: Becoming Leader. State: Replica: d3dfdbcc949c49839481eebfb80e05d7, State: Running, Role: LEADER
I20260620 11:45:16.117331 22463 sys_catalog.cc:564] T 00000000000000000000000000000000 P d3dfdbcc949c49839481eebfb80e05d7 [sys.catalog]: configured and running, proceeding with master startup.
I20260620 11:45:16.117375 22466 consensus_queue.cc:237] T 00000000000000000000000000000000 P d3dfdbcc949c49839481eebfb80e05d7 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d3dfdbcc949c49839481eebfb80e05d7" member_type: VOTER last_known_addr { host: "127.6.234.254" port: 41033 } }
I20260620 11:45:16.118046 22468 sys_catalog.cc:455] T 00000000000000000000000000000000 P d3dfdbcc949c49839481eebfb80e05d7 [sys.catalog]: SysCatalogTable state changed. Reason: New leader d3dfdbcc949c49839481eebfb80e05d7. Latest consensus state: current_term: 1 leader_uuid: "d3dfdbcc949c49839481eebfb80e05d7" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d3dfdbcc949c49839481eebfb80e05d7" member_type: VOTER last_known_addr { host: "127.6.234.254" port: 41033 } } }
I20260620 11:45:16.118026 22467 sys_catalog.cc:455] T 00000000000000000000000000000000 P d3dfdbcc949c49839481eebfb80e05d7 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "d3dfdbcc949c49839481eebfb80e05d7" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d3dfdbcc949c49839481eebfb80e05d7" member_type: VOTER last_known_addr { host: "127.6.234.254" port: 41033 } } }
I20260620 11:45:16.118196 22468 sys_catalog.cc:458] T 00000000000000000000000000000000 P d3dfdbcc949c49839481eebfb80e05d7 [sys.catalog]: This master's current role is: LEADER
I20260620 11:45:16.118201 22467 sys_catalog.cc:458] T 00000000000000000000000000000000 P d3dfdbcc949c49839481eebfb80e05d7 [sys.catalog]: This master's current role is: LEADER
I20260620 11:45:16.118443 22475 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260620 11:45:16.118985 22475 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260620 11:45:16.119870 7083 external_mini_cluster.cc:1388] Started /tmp/dist-test-task1JRg7N/build/release/bin/kudu as pid 22395
I20260620 11:45:16.119988 7083 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task1JRg7N/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781955746769051-7083-0/minicluster-data/master-0/wal/instance
I20260620 11:45:16.120188 7083 external_mini_cluster.cc:1428] Setting key fac112a401b8769acb3f8bc0b94ef15b
I20260620 11:45:16.120788 22475 catalog_manager.cc:1347] Generated new cluster ID: 5867141547be4785bce730c8f383fa7c
I20260620 11:45:16.120838 22475 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260620 11:45:16.126273 22475 catalog_manager.cc:1370] Generated new certificate authority record
I20260620 11:45:16.127048 22475 catalog_manager.cc:1504] Loading token signing keys...
Jun 20 11:45:16 dist-test-slave-cjc6 krb5kdc[22382](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781955915, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.6.234.254@KRBTEST.COM
I20260620 11:45:16.136703 22486 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 11:45:16.120685 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33772 (local address 127.6.234.254:41033)
0620 11:45:16.125948 (+ 5263us) server_negotiation.cc:206] Beginning negotiation
0620 11:45:16.125958 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0620 11:45:16.125994 (+ 36us) server_negotiation.cc:407] Connection header received
0620 11:45:16.126125 (+ 131us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 11:45:16.126133 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 11:45:16.126188 (+ 55us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 11:45:16.126335 (+ 147us) server_negotiation.cc:226] Negotiated authn=SASL
0620 11:45:16.127118 (+ 783us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 11:45:16.128514 (+ 1396us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 11:45:16.129556 (+ 1042us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 11:45:16.130217 (+ 661us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 11:45:16.130332 (+ 115us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 11:45:16.133123 (+ 2791us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 11:45:16.133127 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 11:45:16.133127 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 11:45:16.133139 (+ 12us) server_negotiation.cc:922] calling sasl_server_start()
0620 11:45:16.135089 (+ 1950us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 11:45:16.135500 (+ 411us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 11:45:16.135503 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 11:45:16.135503 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 11:45:16.135601 (+ 98us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 11:45:16.135813 (+ 212us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 11:45:16.135814 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 11:45:16.135814 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 11:45:16.136079 (+ 265us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 11:45:16.136186 (+ 107us) server_negotiation.cc:1035] Waiting for connection context
0620 11:45:16.136339 (+ 153us) server_negotiation.cc:299] Negotiation successful
0620 11:45:16.136475 (+ 136us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":5212,"thread_start_us":69,"threads_started":1}
I20260620 11:45:16.140535 22475 catalog_manager.cc:5868] T 00000000000000000000000000000000 P d3dfdbcc949c49839481eebfb80e05d7: Generated new TSK 0
I20260620 11:45:16.140708 22475 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.6.234.193@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.6.234.193@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.6.234.193 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task1JRg7N/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781955746769051-7083-0/krb5kdc/kudu_127.6.234.193.keytab.
Entry for principal kudu/127.6.234.193 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task1JRg7N/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781955746769051-7083-0/krb5kdc/kudu_127.6.234.193.keytab.
I20260620 11:45:16.166718 7083 external_mini_cluster.cc:1326] Running /tmp/dist-test-task1JRg7N/build/release/bin/kudu
/tmp/dist-test-task1JRg7N/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task1JRg7N/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781955746769051-7083-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-task1JRg7N/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781955746769051-7083-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task1JRg7N/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781955746769051-7083-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-task1JRg7N/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781955746769051-7083-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.6.234.193:0
--local_ip_for_outbound_sockets=127.6.234.193
--webserver_interface=127.6.234.193
--webserver_port=0
--tserver_master_addrs=127.6.234.254:41033
--keytab_file=/tmp/dist-test-task1JRg7N/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781955746769051-7083-0/krb5kdc/kudu_127.6.234.193.keytab
--principal=kudu/127.6.234.193
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.6.234.212:45931
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task1JRg7N/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781955746769051-7083-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task1JRg7N/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781955746769051-7083-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task1JRg7N/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781955746769051-7083-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 11:45:16.246446 22489 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 11:45:16.246642 22489 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 11:45:16.246665 22489 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 11:45:16.247803 22489 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 11:45:16.247845 22489 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 11:45:16.247871 22489 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.6.234.193
I20260620 11:45:16.249310 22489 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.6.234.212:45931
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task1JRg7N/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781955746769051-7083-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-task1JRg7N/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781955746769051-7083-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task1JRg7N/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781955746769051-7083-0/krb5kdc/kudu_127.6.234.193.keytab
--principal=kudu/127.6.234.193
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.6.234.193:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task1JRg7N/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781955746769051-7083-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.6.234.193
--webserver_port=0
--tserver_master_addrs=127.6.234.254:41033
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.22489
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.6.234.193
--log_dir=/tmp/dist-test-task1JRg7N/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781955746769051-7083-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 20 Jun 2026 10:32:37 UTC on ccycloud.quasar-bgyofv.root.comops.site
build id 4457173
I20260620 11:45:16.249511 22489 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 11:45:16.249778 22489 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 11:45:16.254751 22497 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 11:45:16.254750 22495 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 11:45:16.254751 22494 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 11:45:16.254987 22489 server_base.cc:1034] running on GCE node
I20260620 11:45:16.255255 22489 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 11:45:16.255484 22489 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 11:45:16.256645 22489 hybrid_clock.cc:648] HybridClock initialized: now 1781955916256639 us; error 36 us; skew 500 ppm
Jun 20 11:45:16 dist-test-slave-cjc6 krb5kdc[22382](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781955916, etypes {rep=17 tkt=17 ses=17}, kudu/127.6.234.193@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 11:45:16.259153 22489 init.cc:377] Logged in from keytab as kudu/127.6.234.193@KRBTEST.COM (short username kudu)
I20260620 11:45:16.259579 22489 webserver.cc:460] Webserver started at http://127.6.234.193:43611/ using document root <none> and password file <none>
I20260620 11:45:16.259804 22489 fs_manager.cc:359] Metadata directory not provided
I20260620 11:45:16.259868 22489 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 11:45:16.260015 22489 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 11:45:16.261111 22489 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task1JRg7N/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781955746769051-7083-0/minicluster-data/ts-0/data/instance:
uuid: "801a43964a204435806856f3de89b580"
format_stamp: "Formatted at 2026-06-20 11:45:16 on dist-test-slave-cjc6"
server_key: "cf24ad3d777c29baf2ef0bc5d01c92e7"
server_key_iv: "e03d6a10bd25f1620809f4643d5d419f"
server_key_version: "encryptionkey@0"
I20260620 11:45:16.261461 22489 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task1JRg7N/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781955746769051-7083-0/minicluster-data/ts-0/wal/instance:
uuid: "801a43964a204435806856f3de89b580"
format_stamp: "Formatted at 2026-06-20 11:45:16 on dist-test-slave-cjc6"
server_key: "cf24ad3d777c29baf2ef0bc5d01c92e7"
server_key_iv: "e03d6a10bd25f1620809f4643d5d419f"
server_key_version: "encryptionkey@0"
I20260620 11:45:16.262825 22489 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260620 11:45:16.263573 22504 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 11:45:16.263795 22489 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260620 11:45:16.263892 22489 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task1JRg7N/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781955746769051-7083-0/minicluster-data/ts-0/data,/tmp/dist-test-task1JRg7N/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781955746769051-7083-0/minicluster-data/ts-0/wal
uuid: "801a43964a204435806856f3de89b580"
format_stamp: "Formatted at 2026-06-20 11:45:16 on dist-test-slave-cjc6"
server_key: "cf24ad3d777c29baf2ef0bc5d01c92e7"
server_key_iv: "e03d6a10bd25f1620809f4643d5d419f"
server_key_version: "encryptionkey@0"
I20260620 11:45:16.263972 22489 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task1JRg7N/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781955746769051-7083-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-task1JRg7N/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781955746769051-7083-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-task1JRg7N/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781955746769051-7083-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 11:45:16.276633 22489 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 11:45:16.278482 22489 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 11:45:16.278642 22489 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 11:45:16.278857 22489 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 11:45:16.279135 22489 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 11:45:16.279191 22489 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 11:45:16.279233 22489 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 11:45:16.279261 22489 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 11:45:16.285063 22489 rpc_server.cc:307] RPC server started. Bound to: 127.6.234.193:41119
I20260620 11:45:16.285138 22617 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.6.234.193:41119 every 8 connection(s)
I20260620 11:45:16.285452 22489 server_base.cc:1166] Dumped server information to /tmp/dist-test-task1JRg7N/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781955746769051-7083-0/minicluster-data/ts-0/data/info.pb
Jun 20 11:45:16 dist-test-slave-cjc6 krb5kdc[22382](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781955916, etypes {rep=17 tkt=17 ses=17}, kudu/127.6.234.193@KRBTEST.COM for kudu/127.6.234.254@KRBTEST.COM
I20260620 11:45:16.291508 7083 external_mini_cluster.cc:1388] Started /tmp/dist-test-task1JRg7N/build/release/bin/kudu as pid 22489
I20260620 11:45:16.291635 7083 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task1JRg7N/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781955746769051-7083-0/minicluster-data/ts-0/wal/instance
I20260620 11:45:16.291878 7083 external_mini_cluster.cc:1428] Setting key e50e87175d560390d8c521effa36b8cd
I20260620 11:45:16.295006 22620 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 11:45:16.286196 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.234.254:41033 (local address 127.6.234.193:56203)
0620 11:45:16.286348 (+ 152us) negotiation.cc:107] Waiting for socket to connect
0620 11:45:16.286390 (+ 42us) client_negotiation.cc:174] Beginning negotiation
0620 11:45:16.286847 (+ 457us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 11:45:16.287053 (+ 206us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 11:45:16.287056 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 11:45:16.287613 (+ 557us) client_negotiation.cc:189] Negotiated authn=SASL
0620 11:45:16.287794 (+ 181us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 11:45:16.287799 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 11:45:16.288459 (+ 660us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 11:45:16.288460 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 11:45:16.289233 (+ 773us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 11:45:16.289235 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 11:45:16.289566 (+ 331us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 11:45:16.289566 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 11:45:16.289659 (+ 93us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 11:45:16.290190 (+ 531us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 11:45:16.290198 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0620 11:45:16.291680 (+ 1482us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 11:45:16.293560 (+ 1880us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 11:45:16.293568 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 11:45:16.293571 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0620 11:45:16.293943 (+ 372us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 11:45:16.294179 (+ 236us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 11:45:16.294179 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 11:45:16.294179 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0620 11:45:16.294276 (+ 97us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 11:45:16.294550 (+ 274us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 11:45:16.294553 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 11:45:16.294657 (+ 104us) client_negotiation.cc:769] Sending connection context
0620 11:45:16.294739 (+ 82us) client_negotiation.cc:240] Negotiation successful
0620 11:45:16.294854 (+ 115us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":113,"thread_start_us":85,"threads_started":1}
I20260620 11:45:16.295022 22486 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 11:45:16.286293 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.6.234.193:56203 (local address 127.6.234.254:41033)
0620 11:45:16.286343 (+ 50us) server_negotiation.cc:206] Beginning negotiation
0620 11:45:16.286345 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 11:45:16.286474 (+ 129us) server_negotiation.cc:407] Connection header received
0620 11:45:16.286935 (+ 461us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 11:45:16.286936 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 11:45:16.286945 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 11:45:16.287003 (+ 58us) server_negotiation.cc:226] Negotiated authn=SASL
0620 11:45:16.287883 (+ 880us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 11:45:16.288381 (+ 498us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 11:45:16.289308 (+ 927us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 11:45:16.289494 (+ 186us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 11:45:16.289544 (+ 50us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 11:45:16.291800 (+ 2256us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 11:45:16.291802 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 11:45:16.291802 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 11:45:16.291810 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0620 11:45:16.293474 (+ 1664us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 11:45:16.294035 (+ 561us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 11:45:16.294037 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 11:45:16.294037 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 11:45:16.294107 (+ 70us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 11:45:16.294339 (+ 232us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 11:45:16.294340 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 11:45:16.294340 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 11:45:16.294488 (+ 148us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 11:45:16.294564 (+ 76us) server_negotiation.cc:1035] Waiting for connection context
0620 11:45:16.294785 (+ 221us) server_negotiation.cc:299] Negotiation successful
0620 11:45:16.294883 (+ 98us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":23}
I20260620 11:45:16.295471 22618 heartbeater.cc:347] Connected to a master server at 127.6.234.254:41033
I20260620 11:45:16.295567 22618 heartbeater.cc:464] Registering TS with master...
I20260620 11:45:16.295753 22618 heartbeater.cc:511] Master 127.6.234.254:41033 requested a full tablet report, sending...
I20260620 11:45:16.296311 22427 ts_manager.cc:194] Registered new tserver with Master: 801a43964a204435806856f3de89b580 (127.6.234.193:41119)
I20260620 11:45:16.297288 22427 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.6.234.193@KRBTEST.COM'} at 127.6.234.193:56203
WARNING: no policy specified for kudu/127.6.234.194@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.6.234.194@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.6.234.194 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task1JRg7N/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781955746769051-7083-0/krb5kdc/kudu_127.6.234.194.keytab.
Entry for principal kudu/127.6.234.194 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task1JRg7N/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781955746769051-7083-0/krb5kdc/kudu_127.6.234.194.keytab.
I20260620 11:45:16.316449 7083 external_mini_cluster.cc:1326] Running /tmp/dist-test-task1JRg7N/build/release/bin/kudu
/tmp/dist-test-task1JRg7N/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task1JRg7N/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781955746769051-7083-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-task1JRg7N/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781955746769051-7083-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task1JRg7N/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781955746769051-7083-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-task1JRg7N/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781955746769051-7083-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.6.234.194:0
--local_ip_for_outbound_sockets=127.6.234.194
--webserver_interface=127.6.234.194
--webserver_port=0
--tserver_master_addrs=127.6.234.254:41033
--keytab_file=/tmp/dist-test-task1JRg7N/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781955746769051-7083-0/krb5kdc/kudu_127.6.234.194.keytab
--principal=kudu/127.6.234.194
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.6.234.212:45931
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task1JRg7N/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781955746769051-7083-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task1JRg7N/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781955746769051-7083-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task1JRg7N/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781955746769051-7083-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 11:45:16.397311 22623 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 11:45:16.397542 22623 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 11:45:16.397567 22623 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 11:45:16.398746 22623 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 11:45:16.398789 22623 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 11:45:16.398842 22623 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.6.234.194
I20260620 11:45:16.400254 22623 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.6.234.212:45931
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task1JRg7N/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781955746769051-7083-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-task1JRg7N/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781955746769051-7083-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task1JRg7N/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781955746769051-7083-0/krb5kdc/kudu_127.6.234.194.keytab
--principal=kudu/127.6.234.194
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.6.234.194:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task1JRg7N/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781955746769051-7083-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.6.234.194
--webserver_port=0
--tserver_master_addrs=127.6.234.254:41033
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.22623
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.6.234.194
--log_dir=/tmp/dist-test-task1JRg7N/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781955746769051-7083-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 20 Jun 2026 10:32:37 UTC on ccycloud.quasar-bgyofv.root.comops.site
build id 4457173
I20260620 11:45:16.400504 22623 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 11:45:16.400743 22623 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 11:45:16.405822 22629 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 11:45:16.405829 22631 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 11:45:16.405829 22628 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 11:45:16.405987 22623 server_base.cc:1034] running on GCE node
I20260620 11:45:16.406280 22623 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 11:45:16.406585 22623 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 11:45:16.407750 22623 hybrid_clock.cc:648] HybridClock initialized: now 1781955916407741 us; error 31 us; skew 500 ppm
Jun 20 11:45:16 dist-test-slave-cjc6 krb5kdc[22382](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781955916, etypes {rep=17 tkt=17 ses=17}, kudu/127.6.234.194@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 11:45:16.410327 22623 init.cc:377] Logged in from keytab as kudu/127.6.234.194@KRBTEST.COM (short username kudu)
I20260620 11:45:16.410760 22623 webserver.cc:460] Webserver started at http://127.6.234.194:33035/ using document root <none> and password file <none>
I20260620 11:45:16.411001 22623 fs_manager.cc:359] Metadata directory not provided
I20260620 11:45:16.411070 22623 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 11:45:16.411230 22623 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 11:45:16.412261 22623 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task1JRg7N/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781955746769051-7083-0/minicluster-data/ts-1/data/instance:
uuid: "13b3680a21f84e7e8f86c0d8aad45655"
format_stamp: "Formatted at 2026-06-20 11:45:16 on dist-test-slave-cjc6"
server_key: "8991967b56dcbd41188490e2e53c7cb0"
server_key_iv: "6896706017e3442902c05e3f611b85b0"
server_key_version: "encryptionkey@0"
I20260620 11:45:16.412654 22623 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task1JRg7N/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781955746769051-7083-0/minicluster-data/ts-1/wal/instance:
uuid: "13b3680a21f84e7e8f86c0d8aad45655"
format_stamp: "Formatted at 2026-06-20 11:45:16 on dist-test-slave-cjc6"
server_key: "8991967b56dcbd41188490e2e53c7cb0"
server_key_iv: "6896706017e3442902c05e3f611b85b0"
server_key_version: "encryptionkey@0"
I20260620 11:45:16.413959 22623 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.001s
I20260620 11:45:16.414606 22638 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 11:45:16.414776 22623 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.000s
I20260620 11:45:16.414844 22623 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task1JRg7N/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781955746769051-7083-0/minicluster-data/ts-1/data,/tmp/dist-test-task1JRg7N/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781955746769051-7083-0/minicluster-data/ts-1/wal
uuid: "13b3680a21f84e7e8f86c0d8aad45655"
format_stamp: "Formatted at 2026-06-20 11:45:16 on dist-test-slave-cjc6"
server_key: "8991967b56dcbd41188490e2e53c7cb0"
server_key_iv: "6896706017e3442902c05e3f611b85b0"
server_key_version: "encryptionkey@0"
I20260620 11:45:16.414893 22623 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task1JRg7N/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781955746769051-7083-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-task1JRg7N/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781955746769051-7083-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-task1JRg7N/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781955746769051-7083-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 11:45:16.421573 22623 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 11:45:16.423408 22623 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 11:45:16.423520 22623 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 11:45:16.423710 22623 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 11:45:16.423951 22623 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 11:45:16.423981 22623 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 11:45:16.424005 22623 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 11:45:16.424022 22623 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 11:45:16.429903 22623 rpc_server.cc:307] RPC server started. Bound to: 127.6.234.194:38929
I20260620 11:45:16.429962 22751 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.6.234.194:38929 every 8 connection(s)
I20260620 11:45:16.430263 22623 server_base.cc:1166] Dumped server information to /tmp/dist-test-task1JRg7N/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781955746769051-7083-0/minicluster-data/ts-1/data/info.pb
I20260620 11:45:16.431607 7083 external_mini_cluster.cc:1388] Started /tmp/dist-test-task1JRg7N/build/release/bin/kudu as pid 22623
I20260620 11:45:16.431737 7083 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task1JRg7N/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781955746769051-7083-0/minicluster-data/ts-1/wal/instance
I20260620 11:45:16.431924 7083 external_mini_cluster.cc:1428] Setting key a3bbbc517cf6976b32aebac8cf16569a
Jun 20 11:45:16 dist-test-slave-cjc6 krb5kdc[22382](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781955916, etypes {rep=17 tkt=17 ses=17}, kudu/127.6.234.194@KRBTEST.COM for kudu/127.6.234.254@KRBTEST.COM
WARNING: no policy specified for kudu/127.6.234.195@KRBTEST.COM; defaulting to no policy
I20260620 11:45:16.440181 22486 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 11:45:16.430981 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.6.234.194:34917 (local address 127.6.234.254:41033)
0620 11:45:16.431027 (+ 46us) server_negotiation.cc:206] Beginning negotiation
0620 11:45:16.431029 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 11:45:16.431247 (+ 218us) server_negotiation.cc:407] Connection header received
0620 11:45:16.431856 (+ 609us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 11:45:16.431857 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 11:45:16.431870 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 11:45:16.431933 (+ 63us) server_negotiation.cc:226] Negotiated authn=SASL
0620 11:45:16.433129 (+ 1196us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 11:45:16.433631 (+ 502us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 11:45:16.434660 (+ 1029us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 11:45:16.434859 (+ 199us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 11:45:16.434946 (+ 87us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 11:45:16.437310 (+ 2364us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 11:45:16.437312 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 11:45:16.437312 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 11:45:16.437320 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0620 11:45:16.438784 (+ 1464us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 11:45:16.439200 (+ 416us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 11:45:16.439201 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 11:45:16.439202 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 11:45:16.439263 (+ 61us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 11:45:16.439529 (+ 266us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 11:45:16.439530 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 11:45:16.439530 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 11:45:16.439645 (+ 115us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 11:45:16.439725 (+ 80us) server_negotiation.cc:1035] Waiting for connection context
0620 11:45:16.439993 (+ 268us) server_negotiation.cc:299] Negotiation successful
0620 11:45:16.440077 (+ 84us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":20}
I20260620 11:45:16.440371 22754 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 11:45:16.430942 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.234.254:41033 (local address 127.6.234.194:34917)
0620 11:45:16.431116 (+ 174us) negotiation.cc:107] Waiting for socket to connect
0620 11:45:16.431161 (+ 45us) client_negotiation.cc:174] Beginning negotiation
0620 11:45:16.431757 (+ 596us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 11:45:16.432006 (+ 249us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 11:45:16.432011 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 11:45:16.432832 (+ 821us) client_negotiation.cc:189] Negotiated authn=SASL
0620 11:45:16.433029 (+ 197us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 11:45:16.433035 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 11:45:16.433749 (+ 714us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 11:45:16.433751 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 11:45:16.434566 (+ 815us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 11:45:16.434569 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 11:45:16.434957 (+ 388us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 11:45:16.434958 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 11:45:16.435115 (+ 157us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 11:45:16.435604 (+ 489us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 11:45:16.435615 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0620 11:45:16.437212 (+ 1597us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 11:45:16.438886 (+ 1674us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 11:45:16.438894 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 11:45:16.438896 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0620 11:45:16.439119 (+ 223us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 11:45:16.439332 (+ 213us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 11:45:16.439332 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 11:45:16.439333 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0620 11:45:16.439453 (+ 120us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 11:45:16.439716 (+ 263us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 11:45:16.439722 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 11:45:16.439863 (+ 141us) client_negotiation.cc:769] Sending connection context
0620 11:45:16.439975 (+ 112us) client_negotiation.cc:240] Negotiation successful
0620 11:45:16.440133 (+ 158us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":134,"thread_start_us":82,"threads_started":1}
I20260620 11:45:16.440866 22752 heartbeater.cc:347] Connected to a master server at 127.6.234.254:41033
I20260620 11:45:16.441089 22752 heartbeater.cc:464] Registering TS with master...
I20260620 11:45:16.441296 22752 heartbeater.cc:511] Master 127.6.234.254:41033 requested a full tablet report, sending...
I20260620 11:45:16.441789 22427 ts_manager.cc:194] Registered new tserver with Master: 13b3680a21f84e7e8f86c0d8aad45655 (127.6.234.194:38929)
I20260620 11:45:16.442147 22427 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.6.234.194@KRBTEST.COM'} at 127.6.234.194:34917
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.6.234.195@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.6.234.195 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task1JRg7N/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781955746769051-7083-0/krb5kdc/kudu_127.6.234.195.keytab.
Entry for principal kudu/127.6.234.195 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task1JRg7N/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781955746769051-7083-0/krb5kdc/kudu_127.6.234.195.keytab.
I20260620 11:45:16.457418 7083 external_mini_cluster.cc:1326] Running /tmp/dist-test-task1JRg7N/build/release/bin/kudu
/tmp/dist-test-task1JRg7N/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task1JRg7N/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781955746769051-7083-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-task1JRg7N/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781955746769051-7083-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task1JRg7N/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781955746769051-7083-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-task1JRg7N/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781955746769051-7083-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.6.234.195:0
--local_ip_for_outbound_sockets=127.6.234.195
--webserver_interface=127.6.234.195
--webserver_port=0
--tserver_master_addrs=127.6.234.254:41033
--keytab_file=/tmp/dist-test-task1JRg7N/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781955746769051-7083-0/krb5kdc/kudu_127.6.234.195.keytab
--principal=kudu/127.6.234.195
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.6.234.212:45931
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task1JRg7N/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781955746769051-7083-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task1JRg7N/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781955746769051-7083-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task1JRg7N/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781955746769051-7083-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 11:45:16.538038 22757 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 11:45:16.538234 22757 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 11:45:16.538257 22757 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 11:45:16.539403 22757 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 11:45:16.539445 22757 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 11:45:16.539505 22757 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.6.234.195
I20260620 11:45:16.540912 22757 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.6.234.212:45931
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task1JRg7N/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781955746769051-7083-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-task1JRg7N/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781955746769051-7083-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task1JRg7N/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781955746769051-7083-0/krb5kdc/kudu_127.6.234.195.keytab
--principal=kudu/127.6.234.195
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.6.234.195:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task1JRg7N/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781955746769051-7083-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.6.234.195
--webserver_port=0
--tserver_master_addrs=127.6.234.254:41033
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.22757
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.6.234.195
--log_dir=/tmp/dist-test-task1JRg7N/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781955746769051-7083-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 20 Jun 2026 10:32:37 UTC on ccycloud.quasar-bgyofv.root.comops.site
build id 4457173
I20260620 11:45:16.541159 22757 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 11:45:16.541385 22757 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 11:45:16.546386 22763 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 11:45:16.546358 22762 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 11:45:16.546363 22765 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 11:45:16.546658 22757 server_base.cc:1034] running on GCE node
I20260620 11:45:16.546846 22757 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 11:45:16.547084 22757 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 11:45:16.548241 22757 hybrid_clock.cc:648] HybridClock initialized: now 1781955916548226 us; error 26 us; skew 500 ppm
Jun 20 11:45:16 dist-test-slave-cjc6 krb5kdc[22382](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781955916, etypes {rep=17 tkt=17 ses=17}, kudu/127.6.234.195@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 11:45:16.550645 22757 init.cc:377] Logged in from keytab as kudu/127.6.234.195@KRBTEST.COM (short username kudu)
I20260620 11:45:16.551055 22757 webserver.cc:460] Webserver started at http://127.6.234.195:33353/ using document root <none> and password file <none>
I20260620 11:45:16.551311 22757 fs_manager.cc:359] Metadata directory not provided
I20260620 11:45:16.551378 22757 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 11:45:16.551524 22757 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 11:45:16.552534 22757 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task1JRg7N/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781955746769051-7083-0/minicluster-data/ts-2/data/instance:
uuid: "0771078635204f9c92c6d7affdcd6146"
format_stamp: "Formatted at 2026-06-20 11:45:16 on dist-test-slave-cjc6"
server_key: "51adefad54aaad11cbfeb7878e1de217"
server_key_iv: "bbe22f128939827c5b80d1b362603696"
server_key_version: "encryptionkey@0"
I20260620 11:45:16.552835 22757 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task1JRg7N/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781955746769051-7083-0/minicluster-data/ts-2/wal/instance:
uuid: "0771078635204f9c92c6d7affdcd6146"
format_stamp: "Formatted at 2026-06-20 11:45:16 on dist-test-slave-cjc6"
server_key: "51adefad54aaad11cbfeb7878e1de217"
server_key_iv: "bbe22f128939827c5b80d1b362603696"
server_key_version: "encryptionkey@0"
I20260620 11:45:16.554086 22757 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260620 11:45:16.554777 22772 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 11:45:16.554970 22757 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260620 11:45:16.555061 22757 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task1JRg7N/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781955746769051-7083-0/minicluster-data/ts-2/data,/tmp/dist-test-task1JRg7N/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781955746769051-7083-0/minicluster-data/ts-2/wal
uuid: "0771078635204f9c92c6d7affdcd6146"
format_stamp: "Formatted at 2026-06-20 11:45:16 on dist-test-slave-cjc6"
server_key: "51adefad54aaad11cbfeb7878e1de217"
server_key_iv: "bbe22f128939827c5b80d1b362603696"
server_key_version: "encryptionkey@0"
I20260620 11:45:16.555131 22757 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task1JRg7N/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781955746769051-7083-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-task1JRg7N/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781955746769051-7083-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-task1JRg7N/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781955746769051-7083-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 11:45:16.569830 22757 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 11:45:16.571633 22757 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 11:45:16.571796 22757 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 11:45:16.572057 22757 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 11:45:16.572379 22757 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 11:45:16.572432 22757 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 11:45:16.572468 22757 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 11:45:16.572494 22757 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 11:45:16.578320 22757 rpc_server.cc:307] RPC server started. Bound to: 127.6.234.195:34209
I20260620 11:45:16.578353 22885 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.6.234.195:34209 every 8 connection(s)
I20260620 11:45:16.578756 22757 server_base.cc:1166] Dumped server information to /tmp/dist-test-task1JRg7N/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781955746769051-7083-0/minicluster-data/ts-2/data/info.pb
I20260620 11:45:16.581980 7083 external_mini_cluster.cc:1388] Started /tmp/dist-test-task1JRg7N/build/release/bin/kudu as pid 22757
I20260620 11:45:16.582095 7083 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task1JRg7N/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781955746769051-7083-0/minicluster-data/ts-2/wal/instance
I20260620 11:45:16.582286 7083 external_mini_cluster.cc:1428] Setting key 7b87c5877e80873be1d49dada437c83d
Jun 20 11:45:16 dist-test-slave-cjc6 krb5kdc[22382](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781955916, etypes {rep=17 tkt=17 ses=17}, kudu/127.6.234.195@KRBTEST.COM for kudu/127.6.234.254@KRBTEST.COM
I20260620 11:45:16.587572 22486 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 11:45:16.579485 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.6.234.195:38505 (local address 127.6.234.254:41033)
0620 11:45:16.579552 (+ 67us) server_negotiation.cc:206] Beginning negotiation
0620 11:45:16.579554 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 11:45:16.579699 (+ 145us) server_negotiation.cc:407] Connection header received
0620 11:45:16.580168 (+ 469us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 11:45:16.580169 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 11:45:16.580180 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 11:45:16.580236 (+ 56us) server_negotiation.cc:226] Negotiated authn=SASL
0620 11:45:16.581103 (+ 867us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 11:45:16.581586 (+ 483us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 11:45:16.582879 (+ 1293us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 11:45:16.583215 (+ 336us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 11:45:16.583278 (+ 63us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 11:45:16.585235 (+ 1957us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 11:45:16.585236 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 11:45:16.585237 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 11:45:16.585243 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0620 11:45:16.586475 (+ 1232us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 11:45:16.586815 (+ 340us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 11:45:16.586816 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 11:45:16.586816 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 11:45:16.586865 (+ 49us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 11:45:16.587076 (+ 211us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 11:45:16.587077 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 11:45:16.587077 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 11:45:16.587181 (+ 104us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 11:45:16.587230 (+ 49us) server_negotiation.cc:1035] Waiting for connection context
0620 11:45:16.587446 (+ 216us) server_negotiation.cc:299] Negotiation successful
0620 11:45:16.587511 (+ 65us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":31}
I20260620 11:45:16.587813 22888 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 11:45:16.579430 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.234.254:41033 (local address 127.6.234.195:38505)
0620 11:45:16.579579 (+ 149us) negotiation.cc:107] Waiting for socket to connect
0620 11:45:16.579619 (+ 40us) client_negotiation.cc:174] Beginning negotiation
0620 11:45:16.580089 (+ 470us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 11:45:16.580287 (+ 198us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 11:45:16.580290 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 11:45:16.580833 (+ 543us) client_negotiation.cc:189] Negotiated authn=SASL
0620 11:45:16.581020 (+ 187us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 11:45:16.581024 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 11:45:16.581666 (+ 642us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 11:45:16.581667 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 11:45:16.582786 (+ 1119us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 11:45:16.582789 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 11:45:16.583297 (+ 508us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 11:45:16.583298 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 11:45:16.583405 (+ 107us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 11:45:16.583837 (+ 432us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 11:45:16.583845 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0620 11:45:16.585153 (+ 1308us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 11:45:16.586542 (+ 1389us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 11:45:16.586549 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 11:45:16.586550 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0620 11:45:16.586751 (+ 201us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 11:45:16.586924 (+ 173us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 11:45:16.586925 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 11:45:16.586925 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0620 11:45:16.587022 (+ 97us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 11:45:16.587234 (+ 212us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 11:45:16.587238 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 11:45:16.587341 (+ 103us) client_negotiation.cc:769] Sending connection context
0620 11:45:16.587419 (+ 78us) client_negotiation.cc:240] Negotiation successful
0620 11:45:16.587537 (+ 118us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":110,"thread_start_us":82,"threads_started":1}
I20260620 11:45:16.588397 22886 heartbeater.cc:347] Connected to a master server at 127.6.234.254:41033
I20260620 11:45:16.588505 22886 heartbeater.cc:464] Registering TS with master...
I20260620 11:45:16.588671 22886 heartbeater.cc:511] Master 127.6.234.254:41033 requested a full tablet report, sending...
I20260620 11:45:16.589030 22427 ts_manager.cc:194] Registered new tserver with Master: 0771078635204f9c92c6d7affdcd6146 (127.6.234.195:34209)
I20260620 11:45:16.589391 22427 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.6.234.195@KRBTEST.COM'} at 127.6.234.195:38505
I20260620 11:45:16.594033 7083 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260620 11:45:16.600706 22486 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 11:45:16.595309 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33774 (local address 127.6.234.254:41033)
0620 11:45:16.595376 (+ 67us) server_negotiation.cc:206] Beginning negotiation
0620 11:45:16.595378 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 11:45:16.595463 (+ 85us) server_negotiation.cc:407] Connection header received
0620 11:45:16.595501 (+ 38us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 11:45:16.595501 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 11:45:16.595510 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 11:45:16.595579 (+ 69us) server_negotiation.cc:226] Negotiated authn=SASL
0620 11:45:16.596088 (+ 509us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 11:45:16.596530 (+ 442us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 11:45:16.597246 (+ 716us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 11:45:16.597426 (+ 180us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 11:45:16.597495 (+ 69us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 11:45:16.598635 (+ 1140us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 11:45:16.598637 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 11:45:16.598637 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 11:45:16.598643 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0620 11:45:16.599803 (+ 1160us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 11:45:16.600100 (+ 297us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 11:45:16.600101 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 11:45:16.600102 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 11:45:16.600148 (+ 46us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 11:45:16.600306 (+ 158us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 11:45:16.600307 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 11:45:16.600307 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 11:45:16.600431 (+ 124us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 11:45:16.600529 (+ 98us) server_negotiation.cc:1035] Waiting for connection context
0620 11:45:16.600589 (+ 60us) server_negotiation.cc:299] Negotiation successful
0620 11:45:16.600645 (+ 56us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":40}
I20260620 11:45:16.602393 22427 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:33774:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260620 11:45:16.603915 22427 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260620 11:45:16.610423 22897 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 11:45:16.606295 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38280 (local address 127.6.234.193:41119)
0620 11:45:16.606976 (+ 681us) server_negotiation.cc:206] Beginning negotiation
0620 11:45:16.606980 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 11:45:16.606990 (+ 10us) server_negotiation.cc:407] Connection header received
0620 11:45:16.607026 (+ 36us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 11:45:16.607028 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 11:45:16.607043 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 11:45:16.607132 (+ 89us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 11:45:16.607245 (+ 113us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 11:45:16.608280 (+ 1035us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 11:45:16.609373 (+ 1093us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 11:45:16.609975 (+ 602us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 11:45:16.610072 (+ 97us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 11:45:16.610133 (+ 61us) server_negotiation.cc:1035] Waiting for connection context
0620 11:45:16.610293 (+ 160us) server_negotiation.cc:299] Negotiation successful
0620 11:45:16.610335 (+ 42us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":642,"thread_start_us":97,"threads_started":1}
I20260620 11:45:16.610456 22894 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 11:45:16.606182 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.234.193:41119 (local address 127.0.0.1:38280)
0620 11:45:16.606500 (+ 318us) negotiation.cc:107] Waiting for socket to connect
0620 11:45:16.606559 (+ 59us) client_negotiation.cc:174] Beginning negotiation
0620 11:45:16.606625 (+ 66us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 11:45:16.607141 (+ 516us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 11:45:16.607142 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 11:45:16.607144 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 11:45:16.607183 (+ 39us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 11:45:16.607184 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 11:45:16.608358 (+ 1174us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 11:45:16.608359 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 11:45:16.609285 (+ 926us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 11:45:16.609288 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 11:45:16.610075 (+ 787us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 11:45:16.610076 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 11:45:16.610159 (+ 83us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 11:45:16.610171 (+ 12us) client_negotiation.cc:769] Sending connection context
0620 11:45:16.610273 (+ 102us) client_negotiation.cc:240] Negotiation successful
0620 11:45:16.610327 (+ 54us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":300,"mutex_wait_us":1,"thread_start_us":53,"threads_started":1}
I20260620 11:45:16.610495 22898 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 11:45:16.606350 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.234.195:34209 (local address 127.0.0.1:46518)
0620 11:45:16.606730 (+ 380us) negotiation.cc:107] Waiting for socket to connect
0620 11:45:16.606739 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0620 11:45:16.606778 (+ 39us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 11:45:16.607025 (+ 247us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 11:45:16.607026 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 11:45:16.607028 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 11:45:16.607079 (+ 51us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 11:45:16.607080 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 11:45:16.607963 (+ 883us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 11:45:16.607964 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 11:45:16.609197 (+ 1233us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 11:45:16.609199 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 11:45:16.610200 (+ 1001us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 11:45:16.610202 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 11:45:16.610265 (+ 63us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 11:45:16.610269 (+ 4us) client_negotiation.cc:769] Sending connection context
0620 11:45:16.610317 (+ 48us) client_negotiation.cc:240] Negotiation successful
0620 11:45:16.610358 (+ 41us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":366,"thread_start_us":76,"threads_started":1}
I20260620 11:45:16.610608 22899 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 11:45:16.606446 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46518 (local address 127.6.234.195:34209)
0620 11:45:16.606771 (+ 325us) server_negotiation.cc:206] Beginning negotiation
0620 11:45:16.606777 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0620 11:45:16.606788 (+ 11us) server_negotiation.cc:407] Connection header received
0620 11:45:16.606834 (+ 46us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 11:45:16.606836 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 11:45:16.606856 (+ 20us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 11:45:16.606945 (+ 89us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 11:45:16.607173 (+ 228us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 11:45:16.607859 (+ 686us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 11:45:16.609290 (+ 1431us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 11:45:16.610029 (+ 739us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 11:45:16.610110 (+ 81us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 11:45:16.610163 (+ 53us) server_negotiation.cc:1035] Waiting for connection context
0620 11:45:16.610483 (+ 320us) server_negotiation.cc:299] Negotiation successful
0620 11:45:16.610528 (+ 45us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":158,"thread_start_us":91,"threads_started":1}
I20260620 11:45:16.610720 22896 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 11:45:16.606256 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50476 (local address 127.6.234.194:38929)
0620 11:45:16.606660 (+ 404us) server_negotiation.cc:206] Beginning negotiation
0620 11:45:16.606666 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0620 11:45:16.606679 (+ 13us) server_negotiation.cc:407] Connection header received
0620 11:45:16.606720 (+ 41us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 11:45:16.606722 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 11:45:16.606743 (+ 21us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 11:45:16.606833 (+ 90us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 11:45:16.607058 (+ 225us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 11:45:16.608203 (+ 1145us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 11:45:16.609471 (+ 1268us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 11:45:16.610174 (+ 703us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 11:45:16.610234 (+ 60us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 11:45:16.610280 (+ 46us) server_negotiation.cc:1035] Waiting for connection context
0620 11:45:16.610596 (+ 316us) server_negotiation.cc:299] Negotiation successful
0620 11:45:16.610638 (+ 42us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":368,"thread_start_us":75,"threads_started":1}
I20260620 11:45:16.610759 22895 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 11:45:16.606148 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.234.194:38929 (local address 127.0.0.1:50476)
0620 11:45:16.606499 (+ 351us) negotiation.cc:107] Waiting for socket to connect
0620 11:45:16.606542 (+ 43us) client_negotiation.cc:174] Beginning negotiation
0620 11:45:16.606628 (+ 86us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 11:45:16.606902 (+ 274us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 11:45:16.606906 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 11:45:16.606908 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 11:45:16.606969 (+ 61us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 11:45:16.606971 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 11:45:16.608307 (+ 1336us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 11:45:16.608308 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 11:45:16.609398 (+ 1090us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 11:45:16.609400 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 11:45:16.610310 (+ 910us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 11:45:16.610311 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 11:45:16.610363 (+ 52us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 11:45:16.610365 (+ 2us) client_negotiation.cc:769] Sending connection context
0620 11:45:16.610410 (+ 45us) client_negotiation.cc:240] Negotiation successful
0620 11:45:16.610445 (+ 35us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":322,"mutex_wait_us":30,"thread_start_us":64,"threads_started":1}
I20260620 11:45:16.611275 22820 tablet_service.cc:1467] Processing CreateTablet for tablet 9042244e255e4fe8a281eb1cd1f4d5e6 (DEFAULT_TABLE table=test-table [id=dfe32436722444eea696fa4d7fa20321]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 11:45:16.611611 22820 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 9042244e255e4fe8a281eb1cd1f4d5e6. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 11:45:16.611550 22686 tablet_service.cc:1467] Processing CreateTablet for tablet 9042244e255e4fe8a281eb1cd1f4d5e6 (DEFAULT_TABLE table=test-table [id=dfe32436722444eea696fa4d7fa20321]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 11:45:16.611887 22686 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 9042244e255e4fe8a281eb1cd1f4d5e6. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 11:45:16.612118 22552 tablet_service.cc:1467] Processing CreateTablet for tablet 9042244e255e4fe8a281eb1cd1f4d5e6 (DEFAULT_TABLE table=test-table [id=dfe32436722444eea696fa4d7fa20321]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 11:45:16.612434 22552 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 9042244e255e4fe8a281eb1cd1f4d5e6. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 11:45:16.614771 22900 tablet_bootstrap.cc:492] T 9042244e255e4fe8a281eb1cd1f4d5e6 P 13b3680a21f84e7e8f86c0d8aad45655: Bootstrap starting.
I20260620 11:45:16.615012 22901 tablet_bootstrap.cc:492] T 9042244e255e4fe8a281eb1cd1f4d5e6 P 0771078635204f9c92c6d7affdcd6146: Bootstrap starting.
I20260620 11:45:16.615505 22900 tablet_bootstrap.cc:654] T 9042244e255e4fe8a281eb1cd1f4d5e6 P 13b3680a21f84e7e8f86c0d8aad45655: Neither blocks nor log segments found. Creating new log.
I20260620 11:45:16.615692 22901 tablet_bootstrap.cc:654] T 9042244e255e4fe8a281eb1cd1f4d5e6 P 0771078635204f9c92c6d7affdcd6146: Neither blocks nor log segments found. Creating new log.
I20260620 11:45:16.615837 22902 tablet_bootstrap.cc:492] T 9042244e255e4fe8a281eb1cd1f4d5e6 P 801a43964a204435806856f3de89b580: Bootstrap starting.
I20260620 11:45:16.616217 22901 log.cc:824] T 9042244e255e4fe8a281eb1cd1f4d5e6 P 0771078635204f9c92c6d7affdcd6146: Log is configured to *not* fsync() on all Append() calls
I20260620 11:45:16.616266 22900 log.cc:824] T 9042244e255e4fe8a281eb1cd1f4d5e6 P 13b3680a21f84e7e8f86c0d8aad45655: Log is configured to *not* fsync() on all Append() calls
I20260620 11:45:16.616500 22902 tablet_bootstrap.cc:654] T 9042244e255e4fe8a281eb1cd1f4d5e6 P 801a43964a204435806856f3de89b580: Neither blocks nor log segments found. Creating new log.
I20260620 11:45:16.616977 22901 tablet_bootstrap.cc:492] T 9042244e255e4fe8a281eb1cd1f4d5e6 P 0771078635204f9c92c6d7affdcd6146: No bootstrap required, opened a new log
I20260620 11:45:16.617012 22902 log.cc:824] T 9042244e255e4fe8a281eb1cd1f4d5e6 P 801a43964a204435806856f3de89b580: Log is configured to *not* fsync() on all Append() calls
I20260620 11:45:16.617071 22901 ts_tablet_manager.cc:1397] T 9042244e255e4fe8a281eb1cd1f4d5e6 P 0771078635204f9c92c6d7affdcd6146: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20260620 11:45:16.617508 22900 tablet_bootstrap.cc:492] T 9042244e255e4fe8a281eb1cd1f4d5e6 P 13b3680a21f84e7e8f86c0d8aad45655: No bootstrap required, opened a new log
I20260620 11:45:16.617640 22900 ts_tablet_manager.cc:1397] T 9042244e255e4fe8a281eb1cd1f4d5e6 P 13b3680a21f84e7e8f86c0d8aad45655: Time spent bootstrapping tablet: real 0.003s user 0.002s sys 0.000s
I20260620 11:45:16.617830 22902 tablet_bootstrap.cc:492] T 9042244e255e4fe8a281eb1cd1f4d5e6 P 801a43964a204435806856f3de89b580: No bootstrap required, opened a new log
I20260620 11:45:16.617898 22902 ts_tablet_manager.cc:1397] T 9042244e255e4fe8a281eb1cd1f4d5e6 P 801a43964a204435806856f3de89b580: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20260620 11:45:16.619351 22901 raft_consensus.cc:348] T 9042244e255e4fe8a281eb1cd1f4d5e6 P 0771078635204f9c92c6d7affdcd6146 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "13b3680a21f84e7e8f86c0d8aad45655" member_type: VOTER last_known_addr { host: "127.6.234.194" port: 38929 } } peers { permanent_uuid: "0771078635204f9c92c6d7affdcd6146" member_type: VOTER last_known_addr { host: "127.6.234.195" port: 34209 } } peers { permanent_uuid: "801a43964a204435806856f3de89b580" member_type: VOTER last_known_addr { host: "127.6.234.193" port: 41119 } }
I20260620 11:45:16.619387 22900 raft_consensus.cc:348] T 9042244e255e4fe8a281eb1cd1f4d5e6 P 13b3680a21f84e7e8f86c0d8aad45655 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "13b3680a21f84e7e8f86c0d8aad45655" member_type: VOTER last_known_addr { host: "127.6.234.194" port: 38929 } } peers { permanent_uuid: "0771078635204f9c92c6d7affdcd6146" member_type: VOTER last_known_addr { host: "127.6.234.195" port: 34209 } } peers { permanent_uuid: "801a43964a204435806856f3de89b580" member_type: VOTER last_known_addr { host: "127.6.234.193" port: 41119 } }
I20260620 11:45:16.619529 22901 raft_consensus.cc:374] T 9042244e255e4fe8a281eb1cd1f4d5e6 P 0771078635204f9c92c6d7affdcd6146 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 11:45:16.619550 22900 raft_consensus.cc:374] T 9042244e255e4fe8a281eb1cd1f4d5e6 P 13b3680a21f84e7e8f86c0d8aad45655 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 11:45:16.619565 22901 raft_consensus.cc:729] T 9042244e255e4fe8a281eb1cd1f4d5e6 P 0771078635204f9c92c6d7affdcd6146 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0771078635204f9c92c6d7affdcd6146, State: Initialized, Role: FOLLOWER
I20260620 11:45:16.619594 22900 raft_consensus.cc:729] T 9042244e255e4fe8a281eb1cd1f4d5e6 P 13b3680a21f84e7e8f86c0d8aad45655 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 13b3680a21f84e7e8f86c0d8aad45655, State: Initialized, Role: FOLLOWER
I20260620 11:45:16.619692 22901 consensus_queue.cc:260] T 9042244e255e4fe8a281eb1cd1f4d5e6 P 0771078635204f9c92c6d7affdcd6146 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "13b3680a21f84e7e8f86c0d8aad45655" member_type: VOTER last_known_addr { host: "127.6.234.194" port: 38929 } } peers { permanent_uuid: "0771078635204f9c92c6d7affdcd6146" member_type: VOTER last_known_addr { host: "127.6.234.195" port: 34209 } } peers { permanent_uuid: "801a43964a204435806856f3de89b580" member_type: VOTER last_known_addr { host: "127.6.234.193" port: 41119 } }
I20260620 11:45:16.619743 22900 consensus_queue.cc:260] T 9042244e255e4fe8a281eb1cd1f4d5e6 P 13b3680a21f84e7e8f86c0d8aad45655 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "13b3680a21f84e7e8f86c0d8aad45655" member_type: VOTER last_known_addr { host: "127.6.234.194" port: 38929 } } peers { permanent_uuid: "0771078635204f9c92c6d7affdcd6146" member_type: VOTER last_known_addr { host: "127.6.234.195" port: 34209 } } peers { permanent_uuid: "801a43964a204435806856f3de89b580" member_type: VOTER last_known_addr { host: "127.6.234.193" port: 41119 } }
I20260620 11:45:16.619941 22901 ts_tablet_manager.cc:1428] T 9042244e255e4fe8a281eb1cd1f4d5e6 P 0771078635204f9c92c6d7affdcd6146: Time spent starting tablet: real 0.003s user 0.003s sys 0.000s
I20260620 11:45:16.620074 22886 heartbeater.cc:503] Master 127.6.234.254:41033 was elected leader, sending a full tablet report...
I20260620 11:45:16.620213 22902 raft_consensus.cc:348] T 9042244e255e4fe8a281eb1cd1f4d5e6 P 801a43964a204435806856f3de89b580 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "13b3680a21f84e7e8f86c0d8aad45655" member_type: VOTER last_known_addr { host: "127.6.234.194" port: 38929 } } peers { permanent_uuid: "0771078635204f9c92c6d7affdcd6146" member_type: VOTER last_known_addr { host: "127.6.234.195" port: 34209 } } peers { permanent_uuid: "801a43964a204435806856f3de89b580" member_type: VOTER last_known_addr { host: "127.6.234.193" port: 41119 } }
I20260620 11:45:16.620322 22902 raft_consensus.cc:374] T 9042244e255e4fe8a281eb1cd1f4d5e6 P 801a43964a204435806856f3de89b580 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 11:45:16.620342 22902 raft_consensus.cc:729] T 9042244e255e4fe8a281eb1cd1f4d5e6 P 801a43964a204435806856f3de89b580 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 801a43964a204435806856f3de89b580, State: Initialized, Role: FOLLOWER
I20260620 11:45:16.620015 22900 ts_tablet_manager.cc:1428] T 9042244e255e4fe8a281eb1cd1f4d5e6 P 13b3680a21f84e7e8f86c0d8aad45655: Time spent starting tablet: real 0.002s user 0.000s sys 0.002s
I20260620 11:45:16.620364 22752 heartbeater.cc:503] Master 127.6.234.254:41033 was elected leader, sending a full tablet report...
I20260620 11:45:16.620441 22902 consensus_queue.cc:260] T 9042244e255e4fe8a281eb1cd1f4d5e6 P 801a43964a204435806856f3de89b580 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "13b3680a21f84e7e8f86c0d8aad45655" member_type: VOTER last_known_addr { host: "127.6.234.194" port: 38929 } } peers { permanent_uuid: "0771078635204f9c92c6d7affdcd6146" member_type: VOTER last_known_addr { host: "127.6.234.195" port: 34209 } } peers { permanent_uuid: "801a43964a204435806856f3de89b580" member_type: VOTER last_known_addr { host: "127.6.234.193" port: 41119 } }
I20260620 11:45:16.620690 22902 ts_tablet_manager.cc:1428] T 9042244e255e4fe8a281eb1cd1f4d5e6 P 801a43964a204435806856f3de89b580: Time spent starting tablet: real 0.003s user 0.003s sys 0.000s
I20260620 11:45:16.620738 22618 heartbeater.cc:503] Master 127.6.234.254:41033 was elected leader, sending a full tablet report...
I20260620 11:45:16.659799 22908 raft_consensus.cc:482] T 9042244e255e4fe8a281eb1cd1f4d5e6 P 801a43964a204435806856f3de89b580 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260620 11:45:16.659953 22908 raft_consensus.cc:504] T 9042244e255e4fe8a281eb1cd1f4d5e6 P 801a43964a204435806856f3de89b580 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "13b3680a21f84e7e8f86c0d8aad45655" member_type: VOTER last_known_addr { host: "127.6.234.194" port: 38929 } } peers { permanent_uuid: "0771078635204f9c92c6d7affdcd6146" member_type: VOTER last_known_addr { host: "127.6.234.195" port: 34209 } } peers { permanent_uuid: "801a43964a204435806856f3de89b580" member_type: VOTER last_known_addr { host: "127.6.234.193" port: 41119 } }
I20260620 11:45:16.660349 22908 leader_election.cc:288] T 9042244e255e4fe8a281eb1cd1f4d5e6 P 801a43964a204435806856f3de89b580 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 13b3680a21f84e7e8f86c0d8aad45655 (127.6.234.194:38929), 0771078635204f9c92c6d7affdcd6146 (127.6.234.195:34209)
I20260620 11:45:16.664145 22896 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 11:45:16.660636 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.6.234.193:34163 (local address 127.6.234.194:38929)
0620 11:45:16.660706 (+ 70us) server_negotiation.cc:206] Beginning negotiation
0620 11:45:16.660708 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 11:45:16.660719 (+ 11us) server_negotiation.cc:407] Connection header received
0620 11:45:16.660755 (+ 36us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 11:45:16.660755 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 11:45:16.660779 (+ 24us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 11:45:16.660856 (+ 77us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 11:45:16.661036 (+ 180us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 11:45:16.661840 (+ 804us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 11:45:16.663358 (+ 1518us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 11:45:16.663763 (+ 405us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 11:45:16.663854 (+ 91us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 11:45:16.663865 (+ 11us) server_negotiation.cc:1035] Waiting for connection context
0620 11:45:16.664016 (+ 151us) server_negotiation.cc:299] Negotiation successful
0620 11:45:16.664070 (+ 54us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":45}
I20260620 11:45:16.664160 22620 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 11:45:16.660529 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.234.194:38929 (local address 127.6.234.193:34163)
0620 11:45:16.660593 (+ 64us) negotiation.cc:107] Waiting for socket to connect
0620 11:45:16.660605 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0620 11:45:16.660651 (+ 46us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 11:45:16.660877 (+ 226us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 11:45:16.660877 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 11:45:16.660879 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 11:45:16.660956 (+ 77us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 11:45:16.660959 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 11:45:16.661926 (+ 967us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 11:45:16.661928 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 11:45:16.663236 (+ 1308us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 11:45:16.663239 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 11:45:16.663854 (+ 615us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 11:45:16.663855 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 11:45:16.663947 (+ 92us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 11:45:16.663951 (+ 4us) client_negotiation.cc:769] Sending connection context
0620 11:45:16.664007 (+ 56us) client_negotiation.cc:240] Negotiation successful
0620 11:45:16.664054 (+ 47us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":41,"mutex_wait_us":45}
I20260620 11:45:16.664181 22899 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 11:45:16.660746 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.6.234.193:57955 (local address 127.6.234.195:34209)
0620 11:45:16.660812 (+ 66us) server_negotiation.cc:206] Beginning negotiation
0620 11:45:16.660814 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 11:45:16.660911 (+ 97us) server_negotiation.cc:407] Connection header received
0620 11:45:16.660948 (+ 37us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 11:45:16.660949 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 11:45:16.660956 (+ 7us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 11:45:16.661024 (+ 68us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 11:45:16.661146 (+ 122us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 11:45:16.661880 (+ 734us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 11:45:16.663370 (+ 1490us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 11:45:16.663764 (+ 394us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 11:45:16.663855 (+ 91us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 11:45:16.663865 (+ 10us) server_negotiation.cc:1035] Waiting for connection context
0620 11:45:16.664074 (+ 209us) server_negotiation.cc:299] Negotiation successful
0620 11:45:16.664121 (+ 47us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":40}
I20260620 11:45:16.664367 22909 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 11:45:16.660684 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.234.195:34209 (local address 127.6.234.193:57955)
0620 11:45:16.660841 (+ 157us) negotiation.cc:107] Waiting for socket to connect
0620 11:45:16.660850 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0620 11:45:16.660904 (+ 54us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 11:45:16.661034 (+ 130us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 11:45:16.661034 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 11:45:16.661035 (+ 1us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 11:45:16.661092 (+ 57us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 11:45:16.661093 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 11:45:16.661949 (+ 856us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 11:45:16.661950 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 11:45:16.663257 (+ 1307us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 11:45:16.663258 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 11:45:16.663869 (+ 611us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 11:45:16.663871 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 11:45:16.663957 (+ 86us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 11:45:16.663961 (+ 4us) client_negotiation.cc:769] Sending connection context
0620 11:45:16.664057 (+ 96us) client_negotiation.cc:240] Negotiation successful
0620 11:45:16.664095 (+ 38us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":137,"mutex_wait_us":68,"thread_start_us":97,"threads_started":1}
I20260620 11:45:16.664538 22706 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9042244e255e4fe8a281eb1cd1f4d5e6" candidate_uuid: "801a43964a204435806856f3de89b580" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "13b3680a21f84e7e8f86c0d8aad45655" is_pre_election: true
I20260620 11:45:16.664744 22706 raft_consensus.cc:2454] T 9042244e255e4fe8a281eb1cd1f4d5e6 P 13b3680a21f84e7e8f86c0d8aad45655 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 801a43964a204435806856f3de89b580 in term 0.
I20260620 11:45:16.664685 22840 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9042244e255e4fe8a281eb1cd1f4d5e6" candidate_uuid: "801a43964a204435806856f3de89b580" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0771078635204f9c92c6d7affdcd6146" is_pre_election: true
I20260620 11:45:16.664867 22840 raft_consensus.cc:2454] T 9042244e255e4fe8a281eb1cd1f4d5e6 P 0771078635204f9c92c6d7affdcd6146 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 801a43964a204435806856f3de89b580 in term 0.
I20260620 11:45:16.664983 22506 leader_election.cc:302] T 9042244e255e4fe8a281eb1cd1f4d5e6 P 801a43964a204435806856f3de89b580 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 13b3680a21f84e7e8f86c0d8aad45655, 801a43964a204435806856f3de89b580; no voters:
I20260620 11:45:16.665131 22908 raft_consensus.cc:2781] T 9042244e255e4fe8a281eb1cd1f4d5e6 P 801a43964a204435806856f3de89b580 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260620 11:45:16.665198 22908 raft_consensus.cc:482] T 9042244e255e4fe8a281eb1cd1f4d5e6 P 801a43964a204435806856f3de89b580 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260620 11:45:16.665223 22908 raft_consensus.cc:3037] T 9042244e255e4fe8a281eb1cd1f4d5e6 P 801a43964a204435806856f3de89b580 [term 0 FOLLOWER]: Advancing to term 1
I20260620 11:45:16.666008 22908 raft_consensus.cc:504] T 9042244e255e4fe8a281eb1cd1f4d5e6 P 801a43964a204435806856f3de89b580 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "13b3680a21f84e7e8f86c0d8aad45655" member_type: VOTER last_known_addr { host: "127.6.234.194" port: 38929 } } peers { permanent_uuid: "0771078635204f9c92c6d7affdcd6146" member_type: VOTER last_known_addr { host: "127.6.234.195" port: 34209 } } peers { permanent_uuid: "801a43964a204435806856f3de89b580" member_type: VOTER last_known_addr { host: "127.6.234.193" port: 41119 } }
I20260620 11:45:16.666142 22908 leader_election.cc:288] T 9042244e255e4fe8a281eb1cd1f4d5e6 P 801a43964a204435806856f3de89b580 [CANDIDATE]: Term 1 election: Requested vote from peers 13b3680a21f84e7e8f86c0d8aad45655 (127.6.234.194:38929), 0771078635204f9c92c6d7affdcd6146 (127.6.234.195:34209)
I20260620 11:45:16.666307 22840 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9042244e255e4fe8a281eb1cd1f4d5e6" candidate_uuid: "801a43964a204435806856f3de89b580" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0771078635204f9c92c6d7affdcd6146"
I20260620 11:45:16.666325 22706 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9042244e255e4fe8a281eb1cd1f4d5e6" candidate_uuid: "801a43964a204435806856f3de89b580" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "13b3680a21f84e7e8f86c0d8aad45655"
I20260620 11:45:16.666401 22840 raft_consensus.cc:3037] T 9042244e255e4fe8a281eb1cd1f4d5e6 P 0771078635204f9c92c6d7affdcd6146 [term 0 FOLLOWER]: Advancing to term 1
I20260620 11:45:16.666399 22706 raft_consensus.cc:3037] T 9042244e255e4fe8a281eb1cd1f4d5e6 P 13b3680a21f84e7e8f86c0d8aad45655 [term 0 FOLLOWER]: Advancing to term 1
I20260620 11:45:16.667142 22706 raft_consensus.cc:2454] T 9042244e255e4fe8a281eb1cd1f4d5e6 P 13b3680a21f84e7e8f86c0d8aad45655 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 801a43964a204435806856f3de89b580 in term 1.
I20260620 11:45:16.667250 22840 raft_consensus.cc:2454] T 9042244e255e4fe8a281eb1cd1f4d5e6 P 0771078635204f9c92c6d7affdcd6146 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 801a43964a204435806856f3de89b580 in term 1.
I20260620 11:45:16.667312 22506 leader_election.cc:302] T 9042244e255e4fe8a281eb1cd1f4d5e6 P 801a43964a204435806856f3de89b580 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 13b3680a21f84e7e8f86c0d8aad45655, 801a43964a204435806856f3de89b580; no voters:
I20260620 11:45:16.667443 22908 raft_consensus.cc:2781] T 9042244e255e4fe8a281eb1cd1f4d5e6 P 801a43964a204435806856f3de89b580 [term 1 FOLLOWER]: Leader election won for term 1
I20260620 11:45:16.667618 22908 raft_consensus.cc:686] T 9042244e255e4fe8a281eb1cd1f4d5e6 P 801a43964a204435806856f3de89b580 [term 1 LEADER]: Becoming Leader. State: Replica: 801a43964a204435806856f3de89b580, State: Running, Role: LEADER
I20260620 11:45:16.667723 22908 consensus_queue.cc:237] T 9042244e255e4fe8a281eb1cd1f4d5e6 P 801a43964a204435806856f3de89b580 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "13b3680a21f84e7e8f86c0d8aad45655" member_type: VOTER last_known_addr { host: "127.6.234.194" port: 38929 } } peers { permanent_uuid: "0771078635204f9c92c6d7affdcd6146" member_type: VOTER last_known_addr { host: "127.6.234.195" port: 34209 } } peers { permanent_uuid: "801a43964a204435806856f3de89b580" member_type: VOTER last_known_addr { host: "127.6.234.193" port: 41119 } }
I20260620 11:45:16.668591 22427 catalog_manager.cc:5495] T 9042244e255e4fe8a281eb1cd1f4d5e6 P 801a43964a204435806856f3de89b580 reported cstate change: term changed from 0 to 1, leader changed from <none> to 801a43964a204435806856f3de89b580 (127.6.234.193). New cstate: current_term: 1 leader_uuid: "801a43964a204435806856f3de89b580" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "13b3680a21f84e7e8f86c0d8aad45655" member_type: VOTER last_known_addr { host: "127.6.234.194" port: 38929 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "0771078635204f9c92c6d7affdcd6146" member_type: VOTER last_known_addr { host: "127.6.234.195" port: 34209 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "801a43964a204435806856f3de89b580" member_type: VOTER last_known_addr { host: "127.6.234.193" port: 41119 } health_report { overall_health: HEALTHY } } }
W20260620 11:45:16.680840 22753 tablet.cc:2369] T 9042244e255e4fe8a281eb1cd1f4d5e6 P 13b3680a21f84e7e8f86c0d8aad45655: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260620 11:45:16.683037 22897 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 11:45:16.680666 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38296 (local address 127.6.234.193:41119)
0620 11:45:16.680707 (+ 41us) server_negotiation.cc:206] Beginning negotiation
0620 11:45:16.680712 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 11:45:16.680724 (+ 12us) server_negotiation.cc:407] Connection header received
0620 11:45:16.680810 (+ 86us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 11:45:16.680811 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 11:45:16.680827 (+ 16us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 11:45:16.680887 (+ 60us) server_negotiation.cc:226] Negotiated authn=TOKEN
0620 11:45:16.681048 (+ 161us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 11:45:16.681520 (+ 472us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 11:45:16.682366 (+ 846us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 11:45:16.682544 (+ 178us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 11:45:16.682590 (+ 46us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 11:45:16.682721 (+ 131us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0620 11:45:16.682816 (+ 95us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0620 11:45:16.682875 (+ 59us) server_negotiation.cc:1035] Waiting for connection context
0620 11:45:16.682968 (+ 93us) server_negotiation.cc:299] Negotiation successful
0620 11:45:16.682985 (+ 17us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":21}
I20260620 11:45:16.684877 22840 raft_consensus.cc:1261] T 9042244e255e4fe8a281eb1cd1f4d5e6 P 0771078635204f9c92c6d7affdcd6146 [term 1 FOLLOWER]: Refusing update from remote peer 801a43964a204435806856f3de89b580: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 11:45:16.684892 22706 raft_consensus.cc:1261] T 9042244e255e4fe8a281eb1cd1f4d5e6 P 13b3680a21f84e7e8f86c0d8aad45655 [term 1 FOLLOWER]: Refusing update from remote peer 801a43964a204435806856f3de89b580: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 11:45:16.685211 22910 consensus_queue.cc:1035] T 9042244e255e4fe8a281eb1cd1f4d5e6 P 801a43964a204435806856f3de89b580 [LEADER]: Connected to new peer: Peer: permanent_uuid: "13b3680a21f84e7e8f86c0d8aad45655" member_type: VOTER last_known_addr { host: "127.6.234.194" port: 38929 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 11:45:16.685359 22908 consensus_queue.cc:1035] T 9042244e255e4fe8a281eb1cd1f4d5e6 P 801a43964a204435806856f3de89b580 [LEADER]: Connected to new peer: Peer: permanent_uuid: "0771078635204f9c92c6d7affdcd6146" member_type: VOTER last_known_addr { host: "127.6.234.195" port: 34209 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 11:45:16.687687 22914 mvcc.cc:205] Tried to move back new op lower bound from 7298891434738388992 to 7298891434671616000. Current Snapshot: MvccSnapshot[applied={T|T < 7298891434738388992}]
I20260620 11:45:16.688314 22915 mvcc.cc:205] Tried to move back new op lower bound from 7298891434738388992 to 7298891434671616000. Current Snapshot: MvccSnapshot[applied={T|T < 7298891434738388992}]
I20260620 11:45:16.689751 22426 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:33774:
table { table_name: "test-table" } modify_external_catalogs: true
I20260620 11:45:16.689851 22426 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:33774:
table { table_name: "test-table" } modify_external_catalogs: true
I20260620 11:45:16.690584 22426 catalog_manager.cc:5782] T 00000000000000000000000000000000 P d3dfdbcc949c49839481eebfb80e05d7: Sending DeleteTablet for 3 replicas of tablet 9042244e255e4fe8a281eb1cd1f4d5e6
I20260620 11:45:16.690936 22552 tablet_service.cc:1514] Processing DeleteTablet for tablet 9042244e255e4fe8a281eb1cd1f4d5e6 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 11:45:16 UTC) from {username='kudu', principal='kudu/127.6.234.254@KRBTEST.COM'} at 127.0.0.1:38280
I20260620 11:45:16.690954 7083 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task1JRg7N/build/release/bin/kudu with pid 22489
I20260620 11:45:16.690964 22820 tablet_service.cc:1514] Processing DeleteTablet for tablet 9042244e255e4fe8a281eb1cd1f4d5e6 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 11:45:16 UTC) from {username='kudu', principal='kudu/127.6.234.254@KRBTEST.COM'} at 127.0.0.1:46518
I20260620 11:45:16.691630 22686 tablet_service.cc:1514] Processing DeleteTablet for tablet 9042244e255e4fe8a281eb1cd1f4d5e6 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 11:45:16 UTC) from {username='kudu', principal='kudu/127.6.234.254@KRBTEST.COM'} at 127.0.0.1:50476
I20260620 11:45:16.691828 22924 tablet_replica.cc:330] T 9042244e255e4fe8a281eb1cd1f4d5e6 P 13b3680a21f84e7e8f86c0d8aad45655: stopping tablet replica
I20260620 11:45:16.691859 22923 tablet_replica.cc:330] T 9042244e255e4fe8a281eb1cd1f4d5e6 P 0771078635204f9c92c6d7affdcd6146: stopping tablet replica
I20260620 11:45:16.691977 22923 raft_consensus.cc:2229] T 9042244e255e4fe8a281eb1cd1f4d5e6 P 0771078635204f9c92c6d7affdcd6146 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260620 11:45:16.691977 22924 raft_consensus.cc:2229] T 9042244e255e4fe8a281eb1cd1f4d5e6 P 13b3680a21f84e7e8f86c0d8aad45655 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260620 11:45:16.692085 22923 raft_consensus.cc:2258] T 9042244e255e4fe8a281eb1cd1f4d5e6 P 0771078635204f9c92c6d7affdcd6146 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 11:45:16.692091 22924 raft_consensus.cc:2258] T 9042244e255e4fe8a281eb1cd1f4d5e6 P 13b3680a21f84e7e8f86c0d8aad45655 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 11:45:16.692543 22923 ts_tablet_manager.cc:1905] T 9042244e255e4fe8a281eb1cd1f4d5e6 P 0771078635204f9c92c6d7affdcd6146: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 11:45:16.693981 22923 ts_tablet_manager.cc:1918] T 9042244e255e4fe8a281eb1cd1f4d5e6 P 0771078635204f9c92c6d7affdcd6146: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260620 11:45:16.694067 22924 ts_tablet_manager.cc:1905] T 9042244e255e4fe8a281eb1cd1f4d5e6 P 13b3680a21f84e7e8f86c0d8aad45655: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 11:45:16.694067 22923 log.cc:1196] T 9042244e255e4fe8a281eb1cd1f4d5e6 P 0771078635204f9c92c6d7affdcd6146: Deleting WAL directory at /tmp/dist-test-task1JRg7N/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781955746769051-7083-0/minicluster-data/ts-2/wal/wals/9042244e255e4fe8a281eb1cd1f4d5e6
I20260620 11:45:16.694391 22923 ts_tablet_manager.cc:1939] T 9042244e255e4fe8a281eb1cd1f4d5e6 P 0771078635204f9c92c6d7affdcd6146: Deleting consensus metadata
I20260620 11:45:16.694809 22412 catalog_manager.cc:4841] TS 0771078635204f9c92c6d7affdcd6146 (127.6.234.195:34209): tablet 9042244e255e4fe8a281eb1cd1f4d5e6 (table test-table [id=dfe32436722444eea696fa4d7fa20321]) successfully deleted
I20260620 11:45:16.695576 22924 ts_tablet_manager.cc:1918] T 9042244e255e4fe8a281eb1cd1f4d5e6 P 13b3680a21f84e7e8f86c0d8aad45655: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260620 11:45:16.695653 22924 log.cc:1196] T 9042244e255e4fe8a281eb1cd1f4d5e6 P 13b3680a21f84e7e8f86c0d8aad45655: Deleting WAL directory at /tmp/dist-test-task1JRg7N/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781955746769051-7083-0/minicluster-data/ts-1/wal/wals/9042244e255e4fe8a281eb1cd1f4d5e6
I20260620 11:45:16.695935 22924 ts_tablet_manager.cc:1939] T 9042244e255e4fe8a281eb1cd1f4d5e6 P 13b3680a21f84e7e8f86c0d8aad45655: Deleting consensus metadata
I20260620 11:45:16.696298 22412 catalog_manager.cc:4841] TS 13b3680a21f84e7e8f86c0d8aad45655 (127.6.234.194:38929): tablet 9042244e255e4fe8a281eb1cd1f4d5e6 (table test-table [id=dfe32436722444eea696fa4d7fa20321]) successfully deleted
W20260620 11:45:16.696676 22411 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.6.234.193:41119 (error 108)
I20260620 11:45:16.696883 7083 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task1JRg7N/build/release/bin/kudu with pid 22623
I20260620 11:45:16.697096 22895 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 11:45:16.696978 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.234.193:41119 (local address 127.0.0.1:38310)
0620 11:45:16.697027 (+ 49us) negotiation.cc:107] Waiting for socket to connect
0620 11:45:16.697060 (+ 33us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.234.193:41119: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":27}
W20260620 11:45:16.697538 22411 catalog_manager.cc:4584] TS 801a43964a204435806856f3de89b580 (127.6.234.193:41119): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 9042244e255e4fe8a281eb1cd1f4d5e6: Network error: Client connection negotiation failed: client connection to 127.6.234.193:41119: connect: Connection refused (error 111)
I20260620 11:45:16.701221 7083 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task1JRg7N/build/release/bin/kudu with pid 22757
I20260620 11:45:16.705473 7083 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task1JRg7N/build/release/bin/kudu with pid 22395
I20260620 11:45:36.710825 7083 external_mini_cluster.cc:1326] Running /tmp/dist-test-task1JRg7N/build/release/bin/kudu
/tmp/dist-test-task1JRg7N/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task1JRg7N/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781955746769051-7083-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task1JRg7N/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781955746769051-7083-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task1JRg7N/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781955746769051-7083-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task1JRg7N/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781955746769051-7083-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.6.234.254:41033
--webserver_interface=127.6.234.254
--webserver_port=42389
--keytab_file=/tmp/dist-test-task1JRg7N/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781955746769051-7083-0/krb5kdc/kudu_127.6.234.254.keytab
--principal=kudu/127.6.234.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.6.234.212:45931
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.6.234.254:41033
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--authn_token_validity_seconds=20
--authz_token_validity_seconds=20
--tsk_rotation_seconds=20
--tsk_private_key_password_cmd=echo foo with env {KRB5CCNAME=/tmp/dist-test-task1JRg7N/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781955746769051-7083-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task1JRg7N/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781955746769051-7083-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task1JRg7N/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781955746769051-7083-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 11:45:36.790827 22926 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 11:45:36.791085 22926 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 11:45:36.791122 22926 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 11:45:36.792265 22926 flags.cc:425] Enabled experimental flag: --authn_token_validity_seconds=20
W20260620 11:45:36.792336 22926 flags.cc:425] Enabled experimental flag: --authz_token_validity_seconds=20
W20260620 11:45:36.792357 22926 flags.cc:425] Enabled experimental flag: --tsk_rotation_seconds=20
W20260620 11:45:36.792375 22926 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 11:45:36.792403 22926 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 11:45:36.792435 22926 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 11:45:36.792464 22926 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 11:45:36.792488 22926 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 11:45:36.793982 22926 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.6.234.212:45931
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task1JRg7N/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781955746769051-7083-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task1JRg7N/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781955746769051-7083-0/minicluster-data/master-0/wal
--authn_token_validity_seconds=20
--authz_token_validity_seconds=20
--tsk_private_key_password_cmd=echo foo
--tsk_rotation_seconds=20
--ipki_ca_key_size=768
--master_addresses=127.6.234.254:41033
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task1JRg7N/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781955746769051-7083-0/krb5kdc/kudu_127.6.234.254.keytab
--principal=kudu/127.6.234.254
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.6.234.254:41033
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task1JRg7N/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781955746769051-7083-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.6.234.254
--webserver_port=42389
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.22926
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task1JRg7N/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781955746769051-7083-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 20 Jun 2026 10:32:37 UTC on ccycloud.quasar-bgyofv.root.comops.site
build id 4457173
I20260620 11:45:36.794263 22926 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 11:45:36.794492 22926 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 11:45:36.803278 22935 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 11:45:36.803272 22926 server_base.cc:1034] running on GCE node
W20260620 11:45:36.803269 22933 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 11:45:36.803278 22932 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 11:45:36.803805 22926 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 11:45:36.804087 22926 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 11:45:36.805303 22926 hybrid_clock.cc:648] HybridClock initialized: now 1781955936805264 us; error 39 us; skew 500 ppm
Jun 20 11:45:36 dist-test-slave-cjc6 krb5kdc[22382](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781955936, etypes {rep=17 tkt=17 ses=17}, kudu/127.6.234.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 11:45:36.811962 22926 init.cc:377] Logged in from keytab as kudu/127.6.234.254@KRBTEST.COM (short username kudu)
I20260620 11:45:36.812431 22926 webserver.cc:460] Webserver started at http://127.6.234.254:42389/ using document root <none> and password file <none>
I20260620 11:45:36.812695 22926 fs_manager.cc:359] Metadata directory not provided
I20260620 11:45:36.812779 22926 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 11:45:36.814271 22926 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.002s sys 0.000s
I20260620 11:45:36.814944 22942 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 11:45:36.815137 22926 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260620 11:45:36.815246 22926 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task1JRg7N/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781955746769051-7083-0/minicluster-data/master-0/data,/tmp/dist-test-task1JRg7N/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781955746769051-7083-0/minicluster-data/master-0/wal
uuid: "d3dfdbcc949c49839481eebfb80e05d7"
format_stamp: "Formatted at 2026-06-20 11:45:16 on dist-test-slave-cjc6"
server_key: "d0eb388e2b925cb0e115a1ea9364db71"
server_key_iv: "03dce555a5f1af3eb04507723e819dc5"
server_key_version: "encryptionkey@0"
I20260620 11:45:36.815675 22926 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task1JRg7N/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781955746769051-7083-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task1JRg7N/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781955746769051-7083-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task1JRg7N/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781955746769051-7083-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 11:45:36.838198 22926 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 11:45:36.840324 22926 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 11:45:36.840512 22926 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 11:45:36.844663 22926 rpc_server.cc:307] RPC server started. Bound to: 127.6.234.254:41033
I20260620 11:45:36.844697 22994 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.6.234.254:41033 every 8 connection(s)
I20260620 11:45:36.845111 22926 server_base.cc:1166] Dumped server information to /tmp/dist-test-task1JRg7N/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781955746769051-7083-0/minicluster-data/master-0/data/info.pb
I20260620 11:45:36.845911 7083 external_mini_cluster.cc:1388] Started /tmp/dist-test-task1JRg7N/build/release/bin/kudu as pid 22926
I20260620 11:45:36.846223 7083 external_mini_cluster.cc:1326] Running /tmp/dist-test-task1JRg7N/build/release/bin/kudu
/tmp/dist-test-task1JRg7N/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task1JRg7N/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781955746769051-7083-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-task1JRg7N/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781955746769051-7083-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task1JRg7N/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781955746769051-7083-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-task1JRg7N/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781955746769051-7083-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.6.234.193:41119
--local_ip_for_outbound_sockets=127.6.234.193
--tserver_master_addrs=127.6.234.254:41033
--webserver_port=43611
--webserver_interface=127.6.234.193
--keytab_file=/tmp/dist-test-task1JRg7N/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781955746769051-7083-0/krb5kdc/kudu_127.6.234.193.keytab
--principal=kudu/127.6.234.193
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.6.234.212:45931
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task1JRg7N/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781955746769051-7083-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task1JRg7N/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781955746769051-7083-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task1JRg7N/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781955746769051-7083-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260620 11:45:36.847226 22995 sys_catalog.cc:263] Verifying existing consensus state
I20260620 11:45:36.848068 22995 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P d3dfdbcc949c49839481eebfb80e05d7: Bootstrap starting.
I20260620 11:45:36.856292 22995 log.cc:824] T 00000000000000000000000000000000 P d3dfdbcc949c49839481eebfb80e05d7: Log is configured to *not* fsync() on all Append() calls
I20260620 11:45:36.858359 22995 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P d3dfdbcc949c49839481eebfb80e05d7: Bootstrap replayed 1/1 log segments. Stats: ops{read=8 overwritten=0 applied=8 ignored=0} inserts{seen=5 ignored=0} mutations{seen=4 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260620 11:45:36.858673 22995 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P d3dfdbcc949c49839481eebfb80e05d7: Bootstrap complete.
I20260620 11:45:36.860963 22995 raft_consensus.cc:348] T 00000000000000000000000000000000 P d3dfdbcc949c49839481eebfb80e05d7 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d3dfdbcc949c49839481eebfb80e05d7" member_type: VOTER last_known_addr { host: "127.6.234.254" port: 41033 } }
I20260620 11:45:36.861186 22995 raft_consensus.cc:729] T 00000000000000000000000000000000 P d3dfdbcc949c49839481eebfb80e05d7 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: d3dfdbcc949c49839481eebfb80e05d7, State: Initialized, Role: FOLLOWER
I20260620 11:45:36.861353 22995 consensus_queue.cc:260] T 00000000000000000000000000000000 P d3dfdbcc949c49839481eebfb80e05d7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 8, Last appended: 1.8, Last appended by leader: 8, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d3dfdbcc949c49839481eebfb80e05d7" member_type: VOTER last_known_addr { host: "127.6.234.254" port: 41033 } }
I20260620 11:45:36.861411 22995 raft_consensus.cc:388] T 00000000000000000000000000000000 P d3dfdbcc949c49839481eebfb80e05d7 [term 1 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260620 11:45:36.861464 22995 raft_consensus.cc:482] T 00000000000000000000000000000000 P d3dfdbcc949c49839481eebfb80e05d7 [term 1 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260620 11:45:36.861510 22995 raft_consensus.cc:3037] T 00000000000000000000000000000000 P d3dfdbcc949c49839481eebfb80e05d7 [term 1 FOLLOWER]: Advancing to term 2
I20260620 11:45:36.862537 22995 raft_consensus.cc:504] T 00000000000000000000000000000000 P d3dfdbcc949c49839481eebfb80e05d7 [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d3dfdbcc949c49839481eebfb80e05d7" member_type: VOTER last_known_addr { host: "127.6.234.254" port: 41033 } }
I20260620 11:45:36.862619 22995 leader_election.cc:302] T 00000000000000000000000000000000 P d3dfdbcc949c49839481eebfb80e05d7 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: d3dfdbcc949c49839481eebfb80e05d7; no voters:
I20260620 11:45:36.862843 22995 leader_election.cc:288] T 00000000000000000000000000000000 P d3dfdbcc949c49839481eebfb80e05d7 [CANDIDATE]: Term 2 election: Requested vote from peers
I20260620 11:45:36.862870 22999 raft_consensus.cc:2781] T 00000000000000000000000000000000 P d3dfdbcc949c49839481eebfb80e05d7 [term 2 FOLLOWER]: Leader election won for term 2
I20260620 11:45:36.862982 22999 raft_consensus.cc:686] T 00000000000000000000000000000000 P d3dfdbcc949c49839481eebfb80e05d7 [term 2 LEADER]: Becoming Leader. State: Replica: d3dfdbcc949c49839481eebfb80e05d7, State: Running, Role: LEADER
I20260620 11:45:36.863068 22999 consensus_queue.cc:237] T 00000000000000000000000000000000 P d3dfdbcc949c49839481eebfb80e05d7 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 1.8, Last appended by leader: 8, Current term: 2, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d3dfdbcc949c49839481eebfb80e05d7" member_type: VOTER last_known_addr { host: "127.6.234.254" port: 41033 } }
I20260620 11:45:36.863103 22995 sys_catalog.cc:564] T 00000000000000000000000000000000 P d3dfdbcc949c49839481eebfb80e05d7 [sys.catalog]: configured and running, proceeding with master startup.
I20260620 11:45:36.863327 22999 sys_catalog.cc:455] T 00000000000000000000000000000000 P d3dfdbcc949c49839481eebfb80e05d7 [sys.catalog]: SysCatalogTable state changed. Reason: New leader d3dfdbcc949c49839481eebfb80e05d7. Latest consensus state: current_term: 2 leader_uuid: "d3dfdbcc949c49839481eebfb80e05d7" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d3dfdbcc949c49839481eebfb80e05d7" member_type: VOTER last_known_addr { host: "127.6.234.254" port: 41033 } } }
I20260620 11:45:36.863413 22999 sys_catalog.cc:458] T 00000000000000000000000000000000 P d3dfdbcc949c49839481eebfb80e05d7 [sys.catalog]: This master's current role is: LEADER
I20260620 11:45:36.863332 23000 sys_catalog.cc:455] T 00000000000000000000000000000000 P d3dfdbcc949c49839481eebfb80e05d7 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 2 leader_uuid: "d3dfdbcc949c49839481eebfb80e05d7" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d3dfdbcc949c49839481eebfb80e05d7" member_type: VOTER last_known_addr { host: "127.6.234.254" port: 41033 } } }
I20260620 11:45:36.863600 23000 sys_catalog.cc:458] T 00000000000000000000000000000000 P d3dfdbcc949c49839481eebfb80e05d7 [sys.catalog]: This master's current role is: LEADER
I20260620 11:45:36.863903 23003 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260620 11:45:36.864668 23003 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260620 11:45:36.864776 23003 catalog_manager.cc:1259] Loaded cluster ID: 5867141547be4785bce730c8f383fa7c
I20260620 11:45:36.865028 23003 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260620 11:45:36.866253 23003 catalog_manager.cc:1504] Loading token signing keys...
I20260620 11:45:36.866437 23003 catalog_manager.cc:5879] T 00000000000000000000000000000000 P d3dfdbcc949c49839481eebfb80e05d7: Loaded TSK: 0
I20260620 11:45:36.874617 23003 catalog_manager.cc:5868] T 00000000000000000000000000000000 P d3dfdbcc949c49839481eebfb80e05d7: Generated new TSK 1
I20260620 11:45:36.874713 23003 catalog_manager.cc:1514] Initializing in-progress tserver states...
W20260620 11:45:36.930792 22997 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 11:45:36.931008 22997 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 11:45:36.931032 22997 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 11:45:36.932183 22997 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 11:45:36.932224 22997 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 11:45:36.932250 22997 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.6.234.193
I20260620 11:45:36.933717 22997 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.6.234.212:45931
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task1JRg7N/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781955746769051-7083-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-task1JRg7N/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781955746769051-7083-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task1JRg7N/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781955746769051-7083-0/krb5kdc/kudu_127.6.234.193.keytab
--principal=kudu/127.6.234.193
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.6.234.193:41119
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task1JRg7N/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781955746769051-7083-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.6.234.193
--webserver_port=43611
--tserver_master_addrs=127.6.234.254:41033
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.22997
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.6.234.193
--log_dir=/tmp/dist-test-task1JRg7N/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781955746769051-7083-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 20 Jun 2026 10:32:37 UTC on ccycloud.quasar-bgyofv.root.comops.site
build id 4457173
I20260620 11:45:36.933962 22997 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 11:45:36.934206 22997 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 11:45:36.940105 23023 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 11:45:36.940105 23022 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 11:45:36.940121 23025 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 11:45:36.940327 22997 server_base.cc:1034] running on GCE node
I20260620 11:45:36.940613 22997 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 11:45:36.940879 22997 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 11:45:36.942034 22997 hybrid_clock.cc:648] HybridClock initialized: now 1781955936942022 us; error 26 us; skew 500 ppm
Jun 20 11:45:36 dist-test-slave-cjc6 krb5kdc[22382](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781955936, etypes {rep=17 tkt=17 ses=17}, kudu/127.6.234.193@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 11:45:36.944581 22997 init.cc:377] Logged in from keytab as kudu/127.6.234.193@KRBTEST.COM (short username kudu)
I20260620 11:45:36.945014 22997 webserver.cc:460] Webserver started at http://127.6.234.193:43611/ using document root <none> and password file <none>
I20260620 11:45:36.945261 22997 fs_manager.cc:359] Metadata directory not provided
I20260620 11:45:36.945345 22997 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 11:45:36.946848 22997 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.001s sys 0.000s
I20260620 11:45:36.947597 23032 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 11:45:36.947788 22997 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260620 11:45:36.947896 22997 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task1JRg7N/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781955746769051-7083-0/minicluster-data/ts-0/data,/tmp/dist-test-task1JRg7N/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781955746769051-7083-0/minicluster-data/ts-0/wal
uuid: "801a43964a204435806856f3de89b580"
format_stamp: "Formatted at 2026-06-20 11:45:16 on dist-test-slave-cjc6"
server_key: "cf24ad3d777c29baf2ef0bc5d01c92e7"
server_key_iv: "e03d6a10bd25f1620809f4643d5d419f"
server_key_version: "encryptionkey@0"
I20260620 11:45:36.948246 22997 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task1JRg7N/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781955746769051-7083-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-task1JRg7N/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781955746769051-7083-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-task1JRg7N/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781955746769051-7083-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 11:45:36.973899 22997 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 11:45:36.976035 22997 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 11:45:36.976202 22997 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 11:45:36.976440 22997 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 11:45:36.976877 23039 ts_tablet_manager.cc:536] Loading tablet metadata (0/1 complete)
I20260620 11:45:36.978789 22997 ts_tablet_manager.cc:579] Loaded tablet metadata (1 total tablets, 1 live tablets)
I20260620 11:45:36.978857 22997 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.002s user 0.000s sys 0.000s
I20260620 11:45:36.978904 22997 ts_tablet_manager.cc:594] Registering tablets (0/1 complete)
I20260620 11:45:36.979537 22997 ts_tablet_manager.cc:610] Registered 1 tablets
I20260620 11:45:36.979593 22997 ts_tablet_manager.cc:589] Time spent register tablets: real 0.001s user 0.000s sys 0.000s
I20260620 11:45:36.979662 23039 tablet_bootstrap.cc:492] T 9042244e255e4fe8a281eb1cd1f4d5e6 P 801a43964a204435806856f3de89b580: Bootstrap starting.
I20260620 11:45:36.986605 22997 rpc_server.cc:307] RPC server started. Bound to: 127.6.234.193:41119
I20260620 11:45:36.986742 23146 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.6.234.193:41119 every 8 connection(s)
I20260620 11:45:36.987023 22997 server_base.cc:1166] Dumped server information to /tmp/dist-test-task1JRg7N/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781955746769051-7083-0/minicluster-data/ts-0/data/info.pb
I20260620 11:45:36.990959 23039 log.cc:824] T 9042244e255e4fe8a281eb1cd1f4d5e6 P 801a43964a204435806856f3de89b580: Log is configured to *not* fsync() on all Append() calls
I20260620 11:45:36.991654 7083 external_mini_cluster.cc:1388] Started /tmp/dist-test-task1JRg7N/build/release/bin/kudu as pid 22997
I20260620 11:45:36.991967 7083 external_mini_cluster.cc:1326] Running /tmp/dist-test-task1JRg7N/build/release/bin/kudu
/tmp/dist-test-task1JRg7N/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task1JRg7N/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781955746769051-7083-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-task1JRg7N/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781955746769051-7083-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task1JRg7N/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781955746769051-7083-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-task1JRg7N/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781955746769051-7083-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.6.234.194:38929
--local_ip_for_outbound_sockets=127.6.234.194
--tserver_master_addrs=127.6.234.254:41033
--webserver_port=33035
--webserver_interface=127.6.234.194
--keytab_file=/tmp/dist-test-task1JRg7N/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781955746769051-7083-0/krb5kdc/kudu_127.6.234.194.keytab
--principal=kudu/127.6.234.194
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.6.234.212:45931
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task1JRg7N/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781955746769051-7083-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task1JRg7N/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781955746769051-7083-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task1JRg7N/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781955746769051-7083-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260620 11:45:36.997162 23039 tablet_bootstrap.cc:492] T 9042244e255e4fe8a281eb1cd1f4d5e6 P 801a43964a204435806856f3de89b580: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=1 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260620 11:45:36.997534 23039 tablet_bootstrap.cc:492] T 9042244e255e4fe8a281eb1cd1f4d5e6 P 801a43964a204435806856f3de89b580: Bootstrap complete.
I20260620 11:45:36.998697 23039 ts_tablet_manager.cc:1397] T 9042244e255e4fe8a281eb1cd1f4d5e6 P 801a43964a204435806856f3de89b580: Time spent bootstrapping tablet: real 0.019s user 0.008s sys 0.006s
I20260620 11:45:37.000025 23039 raft_consensus.cc:348] T 9042244e255e4fe8a281eb1cd1f4d5e6 P 801a43964a204435806856f3de89b580 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "13b3680a21f84e7e8f86c0d8aad45655" member_type: VOTER last_known_addr { host: "127.6.234.194" port: 38929 } } peers { permanent_uuid: "0771078635204f9c92c6d7affdcd6146" member_type: VOTER last_known_addr { host: "127.6.234.195" port: 34209 } } peers { permanent_uuid: "801a43964a204435806856f3de89b580" member_type: VOTER last_known_addr { host: "127.6.234.193" port: 41119 } }
I20260620 11:45:37.000314 23039 raft_consensus.cc:729] T 9042244e255e4fe8a281eb1cd1f4d5e6 P 801a43964a204435806856f3de89b580 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 801a43964a204435806856f3de89b580, State: Initialized, Role: FOLLOWER
I20260620 11:45:37.002496 23039 consensus_queue.cc:260] T 9042244e255e4fe8a281eb1cd1f4d5e6 P 801a43964a204435806856f3de89b580 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "13b3680a21f84e7e8f86c0d8aad45655" member_type: VOTER last_known_addr { host: "127.6.234.194" port: 38929 } } peers { permanent_uuid: "0771078635204f9c92c6d7affdcd6146" member_type: VOTER last_known_addr { host: "127.6.234.195" port: 34209 } } peers { permanent_uuid: "801a43964a204435806856f3de89b580" member_type: VOTER last_known_addr { host: "127.6.234.193" port: 41119 } }
I20260620 11:45:37.002867 23039 ts_tablet_manager.cc:1428] T 9042244e255e4fe8a281eb1cd1f4d5e6 P 801a43964a204435806856f3de89b580: Time spent starting tablet: real 0.004s user 0.001s sys 0.002s
Jun 20 11:45:36 dist-test-slave-cjc6 krb5kdc[22382](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781955936, etypes {rep=17 tkt=17 ses=17}, kudu/127.6.234.193@KRBTEST.COM for kudu/127.6.234.254@KRBTEST.COM
I20260620 11:45:37.008183 23150 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 11:45:36.987799 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.6.234.193:36243 (local address 127.6.234.254:41033)
0620 11:45:36.997930 (+ 10131us) server_negotiation.cc:206] Beginning negotiation
0620 11:45:36.997942 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0620 11:45:36.997983 (+ 41us) server_negotiation.cc:407] Connection header received
0620 11:45:36.998119 (+ 136us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 11:45:36.998127 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 11:45:36.998185 (+ 58us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 11:45:36.998362 (+ 177us) server_negotiation.cc:226] Negotiated authn=SASL
0620 11:45:36.999142 (+ 780us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 11:45:37.000129 (+ 987us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 11:45:37.001201 (+ 1072us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 11:45:37.001632 (+ 431us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 11:45:37.001776 (+ 144us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 11:45:37.004559 (+ 2783us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 11:45:37.004564 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 11:45:37.004564 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 11:45:37.004577 (+ 13us) server_negotiation.cc:922] calling sasl_server_start()
0620 11:45:37.006349 (+ 1772us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 11:45:37.006874 (+ 525us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 11:45:37.006877 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 11:45:37.006878 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 11:45:37.006965 (+ 87us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 11:45:37.007274 (+ 309us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 11:45:37.007274 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 11:45:37.007275 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 11:45:37.007506 (+ 231us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 11:45:37.007563 (+ 57us) server_negotiation.cc:1035] Waiting for connection context
0620 11:45:37.007950 (+ 387us) server_negotiation.cc:299] Negotiation successful
0620 11:45:37.008049 (+ 99us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":10075,"thread_start_us":101,"threads_started":1}
I20260620 11:45:37.008293 23149 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 11:45:36.987710 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.234.254:41033 (local address 127.6.234.193:36243)
0620 11:45:36.987921 (+ 211us) negotiation.cc:107] Waiting for socket to connect
0620 11:45:36.987968 (+ 47us) client_negotiation.cc:174] Beginning negotiation
0620 11:45:36.988409 (+ 441us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 11:45:36.998462 (+ 10053us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 11:45:36.998470 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 11:45:36.998731 (+ 261us) client_negotiation.cc:189] Negotiated authn=SASL
0620 11:45:36.999011 (+ 280us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 11:45:36.999017 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 11:45:37.000241 (+ 1224us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 11:45:37.000242 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 11:45:37.001105 (+ 863us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 11:45:37.001107 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 11:45:37.001752 (+ 645us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 11:45:37.001754 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 11:45:37.001893 (+ 139us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 11:45:37.002456 (+ 563us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 11:45:37.002464 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0620 11:45:37.004439 (+ 1975us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 11:45:37.006435 (+ 1996us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 11:45:37.006443 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 11:45:37.006445 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0620 11:45:37.006783 (+ 338us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 11:45:37.007052 (+ 269us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 11:45:37.007053 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 11:45:37.007054 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0620 11:45:37.007204 (+ 150us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 11:45:37.007590 (+ 386us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 11:45:37.007598 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 11:45:37.007775 (+ 177us) client_negotiation.cc:769] Sending connection context
0620 11:45:37.007908 (+ 133us) client_negotiation.cc:240] Negotiation successful
0620 11:45:37.008089 (+ 181us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":163,"thread_start_us":105,"threads_started":1}
I20260620 11:45:37.008913 23147 heartbeater.cc:347] Connected to a master server at 127.6.234.254:41033
I20260620 11:45:37.009006 23147 heartbeater.cc:464] Registering TS with master...
I20260620 11:45:37.009171 23147 heartbeater.cc:511] Master 127.6.234.254:41033 requested a full tablet report, sending...
I20260620 11:45:37.009929 22959 ts_manager.cc:194] Registered new tserver with Master: 801a43964a204435806856f3de89b580 (127.6.234.193:41119)
I20260620 11:45:37.011004 22959 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.6.234.193@KRBTEST.COM'} at 127.6.234.193:36243
I20260620 11:45:37.011957 23147 heartbeater.cc:503] Master 127.6.234.254:41033 was elected leader, sending a full tablet report...
Jun 20 11:45:37 dist-test-slave-cjc6 krb5kdc[22382](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781955936, etypes {rep=17 tkt=17 ses=17}, kudu/127.6.234.254@KRBTEST.COM for kudu/127.6.234.193@KRBTEST.COM
I20260620 11:45:37.018682 23155 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 11:45:37.010585 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60910 (local address 127.6.234.193:41119)
0620 11:45:37.010722 (+ 137us) server_negotiation.cc:206] Beginning negotiation
0620 11:45:37.010726 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 11:45:37.010902 (+ 176us) server_negotiation.cc:407] Connection header received
0620 11:45:37.010987 (+ 85us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 11:45:37.010990 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 11:45:37.011005 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 11:45:37.011068 (+ 63us) server_negotiation.cc:226] Negotiated authn=SASL
0620 11:45:37.011602 (+ 534us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 11:45:37.012505 (+ 903us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 11:45:37.013228 (+ 723us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 11:45:37.013579 (+ 351us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 11:45:37.013651 (+ 72us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 11:45:37.015736 (+ 2085us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 11:45:37.015744 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 11:45:37.015744 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 11:45:37.015751 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0620 11:45:37.017420 (+ 1669us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 11:45:37.017801 (+ 381us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 11:45:37.017804 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 11:45:37.017805 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 11:45:37.017868 (+ 63us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 11:45:37.018074 (+ 206us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 11:45:37.018075 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 11:45:37.018075 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 11:45:37.018211 (+ 136us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 11:45:37.018305 (+ 94us) server_negotiation.cc:1035] Waiting for connection context
0620 11:45:37.018505 (+ 200us) server_negotiation.cc:299] Negotiation successful
0620 11:45:37.018594 (+ 89us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":112,"thread_start_us":69,"threads_started":1}
I20260620 11:45:37.018680 23154 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 11:45:37.010516 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.234.193:41119 (local address 127.0.0.1:60910)
0620 11:45:37.010786 (+ 270us) negotiation.cc:107] Waiting for socket to connect
0620 11:45:37.010823 (+ 37us) client_negotiation.cc:174] Beginning negotiation
0620 11:45:37.010897 (+ 74us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 11:45:37.011087 (+ 190us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 11:45:37.011092 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 11:45:37.011462 (+ 370us) client_negotiation.cc:189] Negotiated authn=SASL
0620 11:45:37.011521 (+ 59us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 11:45:37.011523 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 11:45:37.012570 (+ 1047us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 11:45:37.012572 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 11:45:37.013133 (+ 561us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 11:45:37.013135 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 11:45:37.013664 (+ 529us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 11:45:37.013665 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 11:45:37.013779 (+ 114us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 11:45:37.014250 (+ 471us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 11:45:37.014255 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0620 11:45:37.015656 (+ 1401us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 11:45:37.017501 (+ 1845us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 11:45:37.017507 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 11:45:37.017508 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0620 11:45:37.017686 (+ 178us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 11:45:37.017947 (+ 261us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 11:45:37.017948 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 11:45:37.017948 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0620 11:45:37.018011 (+ 63us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 11:45:37.018285 (+ 274us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 11:45:37.018292 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 11:45:37.018342 (+ 50us) client_negotiation.cc:769] Sending connection context
0620 11:45:37.018456 (+ 114us) client_negotiation.cc:240] Negotiation successful
0620 11:45:37.018583 (+ 127us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":242,"thread_start_us":93,"threads_started":1}
I20260620 11:45:37.019060 23081 tablet_service.cc:1514] Processing DeleteTablet for tablet 9042244e255e4fe8a281eb1cd1f4d5e6 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 11:45:16 UTC) from {username='kudu', principal='kudu/127.6.234.254@KRBTEST.COM'} at 127.0.0.1:60910
I20260620 11:45:37.019562 23156 tablet_replica.cc:330] stopping tablet replica
I20260620 11:45:37.019675 23156 raft_consensus.cc:2229] T 9042244e255e4fe8a281eb1cd1f4d5e6 P 801a43964a204435806856f3de89b580 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260620 11:45:37.019743 23156 raft_consensus.cc:2258] T 9042244e255e4fe8a281eb1cd1f4d5e6 P 801a43964a204435806856f3de89b580 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 11:45:37.020068 23156 ts_tablet_manager.cc:1905] T 9042244e255e4fe8a281eb1cd1f4d5e6 P 801a43964a204435806856f3de89b580: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 11:45:37.022661 23156 ts_tablet_manager.cc:1918] T 9042244e255e4fe8a281eb1cd1f4d5e6 P 801a43964a204435806856f3de89b580: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260620 11:45:37.022738 23156 log.cc:1196] T 9042244e255e4fe8a281eb1cd1f4d5e6 P 801a43964a204435806856f3de89b580: Deleting WAL directory at /tmp/dist-test-task1JRg7N/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781955746769051-7083-0/minicluster-data/ts-0/wal/wals/9042244e255e4fe8a281eb1cd1f4d5e6
I20260620 11:45:37.022964 23156 ts_tablet_manager.cc:1939] T 9042244e255e4fe8a281eb1cd1f4d5e6 P 801a43964a204435806856f3de89b580: Deleting consensus metadata
I20260620 11:45:37.023487 22943 catalog_manager.cc:4841] TS 801a43964a204435806856f3de89b580 (127.6.234.193:41119): tablet 9042244e255e4fe8a281eb1cd1f4d5e6 (table test-table [id=dfe32436722444eea696fa4d7fa20321]) successfully deleted
W20260620 11:45:37.083905 23152 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 11:45:37.084103 23152 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 11:45:37.084126 23152 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 11:45:37.085376 23152 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 11:45:37.085424 23152 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 11:45:37.085451 23152 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.6.234.194
I20260620 11:45:37.086948 23152 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.6.234.212:45931
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task1JRg7N/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781955746769051-7083-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-task1JRg7N/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781955746769051-7083-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task1JRg7N/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781955746769051-7083-0/krb5kdc/kudu_127.6.234.194.keytab
--principal=kudu/127.6.234.194
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.6.234.194:38929
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task1JRg7N/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781955746769051-7083-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.6.234.194
--webserver_port=33035
--tserver_master_addrs=127.6.234.254:41033
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.23152
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.6.234.194
--log_dir=/tmp/dist-test-task1JRg7N/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781955746769051-7083-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 20 Jun 2026 10:32:37 UTC on ccycloud.quasar-bgyofv.root.comops.site
build id 4457173
I20260620 11:45:37.087186 23152 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 11:45:37.087419 23152 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260620 11:45:37.092278 23152 server_base.cc:1034] running on GCE node
W20260620 11:45:37.092252 23161 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 11:45:37.092242 23162 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 11:45:37.092303 23164 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 11:45:37.092664 23152 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 11:45:37.092921 23152 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 11:45:37.094102 23152 hybrid_clock.cc:648] HybridClock initialized: now 1781955937094083 us; error 47 us; skew 500 ppm
Jun 20 11:45:37 dist-test-slave-cjc6 krb5kdc[22382](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781955937, etypes {rep=17 tkt=17 ses=17}, kudu/127.6.234.194@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 11:45:37.096582 23152 init.cc:377] Logged in from keytab as kudu/127.6.234.194@KRBTEST.COM (short username kudu)
I20260620 11:45:37.097007 23152 webserver.cc:460] Webserver started at http://127.6.234.194:33035/ using document root <none> and password file <none>
I20260620 11:45:37.097244 23152 fs_manager.cc:359] Metadata directory not provided
I20260620 11:45:37.097385 23152 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 11:45:37.098807 23152 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.001s sys 0.000s
I20260620 11:45:37.101824 23171 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 11:45:37.102051 23152 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.000s
I20260620 11:45:37.102172 23152 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task1JRg7N/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781955746769051-7083-0/minicluster-data/ts-1/data,/tmp/dist-test-task1JRg7N/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781955746769051-7083-0/minicluster-data/ts-1/wal
uuid: "13b3680a21f84e7e8f86c0d8aad45655"
format_stamp: "Formatted at 2026-06-20 11:45:16 on dist-test-slave-cjc6"
server_key: "8991967b56dcbd41188490e2e53c7cb0"
server_key_iv: "6896706017e3442902c05e3f611b85b0"
server_key_version: "encryptionkey@0"
I20260620 11:45:37.102681 23152 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task1JRg7N/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781955746769051-7083-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-task1JRg7N/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781955746769051-7083-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-task1JRg7N/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781955746769051-7083-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 11:45:37.122920 23152 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 11:45:37.124919 23152 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 11:45:37.125033 23152 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 11:45:37.125218 23152 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 11:45:37.125553 23152 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 11:45:37.125587 23152 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 11:45:37.125635 23152 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 11:45:37.125662 23152 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 11:45:37.131589 23152 rpc_server.cc:307] RPC server started. Bound to: 127.6.234.194:38929
I20260620 11:45:37.131672 23284 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.6.234.194:38929 every 8 connection(s)
I20260620 11:45:37.131997 23152 server_base.cc:1166] Dumped server information to /tmp/dist-test-task1JRg7N/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781955746769051-7083-0/minicluster-data/ts-1/data/info.pb
Jun 20 11:45:37 dist-test-slave-cjc6 krb5kdc[22382](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781955937, etypes {rep=17 tkt=17 ses=17}, kudu/127.6.234.194@KRBTEST.COM for kudu/127.6.234.254@KRBTEST.COM
I20260620 11:45:37.140937 23150 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 11:45:37.132672 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.6.234.194:53103 (local address 127.6.234.254:41033)
0620 11:45:37.132769 (+ 97us) server_negotiation.cc:206] Beginning negotiation
0620 11:45:37.132772 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0620 11:45:37.132968 (+ 196us) server_negotiation.cc:407] Connection header received
0620 11:45:37.133443 (+ 475us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 11:45:37.133444 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 11:45:37.133453 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 11:45:37.133507 (+ 54us) server_negotiation.cc:226] Negotiated authn=SASL
0620 11:45:37.134507 (+ 1000us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 11:45:37.135030 (+ 523us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 11:45:37.136003 (+ 973us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 11:45:37.136224 (+ 221us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 11:45:37.136291 (+ 67us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 11:45:37.138421 (+ 2130us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 11:45:37.138422 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 11:45:37.138423 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 11:45:37.138429 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0620 11:45:37.139709 (+ 1280us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 11:45:37.140088 (+ 379us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 11:45:37.140089 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 11:45:37.140089 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 11:45:37.140139 (+ 50us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 11:45:37.140372 (+ 233us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 11:45:37.140373 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 11:45:37.140373 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 11:45:37.140479 (+ 106us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 11:45:37.140536 (+ 57us) server_negotiation.cc:1035] Waiting for connection context
0620 11:45:37.140800 (+ 264us) server_negotiation.cc:299] Negotiation successful
0620 11:45:37.140869 (+ 69us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":54}
I20260620 11:45:37.141161 23287 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 11:45:37.132639 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.234.254:41033 (local address 127.6.234.194:53103)
0620 11:45:37.132808 (+ 169us) negotiation.cc:107] Waiting for socket to connect
0620 11:45:37.132850 (+ 42us) client_negotiation.cc:174] Beginning negotiation
0620 11:45:37.133357 (+ 507us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 11:45:37.133563 (+ 206us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 11:45:37.133566 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 11:45:37.134206 (+ 640us) client_negotiation.cc:189] Negotiated authn=SASL
0620 11:45:37.134401 (+ 195us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 11:45:37.134406 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 11:45:37.135129 (+ 723us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 11:45:37.135130 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 11:45:37.135926 (+ 796us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 11:45:37.135929 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 11:45:37.136300 (+ 371us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 11:45:37.136301 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 11:45:37.136404 (+ 103us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 11:45:37.136928 (+ 524us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 11:45:37.136936 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0620 11:45:37.138335 (+ 1399us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 11:45:37.139796 (+ 1461us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 11:45:37.139802 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 11:45:37.139804 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0620 11:45:37.140022 (+ 218us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 11:45:37.140198 (+ 176us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 11:45:37.140199 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 11:45:37.140199 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0620 11:45:37.140308 (+ 109us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 11:45:37.140563 (+ 255us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 11:45:37.140568 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 11:45:37.140681 (+ 113us) client_negotiation.cc:769] Sending connection context
0620 11:45:37.140768 (+ 87us) client_negotiation.cc:240] Negotiation successful
0620 11:45:37.140892 (+ 124us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":132,"thread_start_us":94,"threads_started":1}
I20260620 11:45:37.141712 23285 heartbeater.cc:347] Connected to a master server at 127.6.234.254:41033
I20260620 11:45:37.141839 23285 heartbeater.cc:464] Registering TS with master...
I20260620 11:45:37.142047 23285 heartbeater.cc:511] Master 127.6.234.254:41033 requested a full tablet report, sending...
I20260620 11:45:37.142035 7083 external_mini_cluster.cc:1388] Started /tmp/dist-test-task1JRg7N/build/release/bin/kudu as pid 23152
I20260620 11:45:37.142383 7083 external_mini_cluster.cc:1326] Running /tmp/dist-test-task1JRg7N/build/release/bin/kudu
/tmp/dist-test-task1JRg7N/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task1JRg7N/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781955746769051-7083-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-task1JRg7N/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781955746769051-7083-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task1JRg7N/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781955746769051-7083-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-task1JRg7N/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781955746769051-7083-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.6.234.195:34209
--local_ip_for_outbound_sockets=127.6.234.195
--tserver_master_addrs=127.6.234.254:41033
--webserver_port=33353
--webserver_interface=127.6.234.195
--keytab_file=/tmp/dist-test-task1JRg7N/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781955746769051-7083-0/krb5kdc/kudu_127.6.234.195.keytab
--principal=kudu/127.6.234.195
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.6.234.212:45931
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task1JRg7N/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781955746769051-7083-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task1JRg7N/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781955746769051-7083-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task1JRg7N/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781955746769051-7083-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260620 11:45:37.142484 22959 ts_manager.cc:194] Registered new tserver with Master: 13b3680a21f84e7e8f86c0d8aad45655 (127.6.234.194:38929)
I20260620 11:45:37.142933 22959 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.6.234.194@KRBTEST.COM'} at 127.6.234.194:53103
W20260620 11:45:37.224098 23288 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 11:45:37.224300 23288 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 11:45:37.224330 23288 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 11:45:37.225467 23288 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 11:45:37.225508 23288 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 11:45:37.225534 23288 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.6.234.195
I20260620 11:45:37.227043 23288 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.6.234.212:45931
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task1JRg7N/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781955746769051-7083-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-task1JRg7N/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781955746769051-7083-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task1JRg7N/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781955746769051-7083-0/krb5kdc/kudu_127.6.234.195.keytab
--principal=kudu/127.6.234.195
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.6.234.195:34209
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task1JRg7N/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781955746769051-7083-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.6.234.195
--webserver_port=33353
--tserver_master_addrs=127.6.234.254:41033
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.23288
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.6.234.195
--log_dir=/tmp/dist-test-task1JRg7N/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781955746769051-7083-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 20 Jun 2026 10:32:37 UTC on ccycloud.quasar-bgyofv.root.comops.site
build id 4457173
I20260620 11:45:37.227291 23288 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 11:45:37.227522 23288 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 11:45:37.232440 23293 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 11:45:37.232440 23296 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 11:45:37.232450 23294 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 11:45:37.232471 23288 server_base.cc:1034] running on GCE node
I20260620 11:45:37.232913 23288 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 11:45:37.233150 23288 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 11:45:37.234311 23288 hybrid_clock.cc:648] HybridClock initialized: now 1781955937234312 us; error 44 us; skew 500 ppm
Jun 20 11:45:37 dist-test-slave-cjc6 krb5kdc[22382](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781955937, etypes {rep=17 tkt=17 ses=17}, kudu/127.6.234.195@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 11:45:37.236917 23288 init.cc:377] Logged in from keytab as kudu/127.6.234.195@KRBTEST.COM (short username kudu)
I20260620 11:45:37.237351 23288 webserver.cc:460] Webserver started at http://127.6.234.195:33353/ using document root <none> and password file <none>
I20260620 11:45:37.237583 23288 fs_manager.cc:359] Metadata directory not provided
I20260620 11:45:37.237666 23288 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 11:45:37.239068 23288 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.001s sys 0.000s
I20260620 11:45:37.239706 23303 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 11:45:37.239883 23288 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260620 11:45:37.239982 23288 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task1JRg7N/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781955746769051-7083-0/minicluster-data/ts-2/data,/tmp/dist-test-task1JRg7N/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781955746769051-7083-0/minicluster-data/ts-2/wal
uuid: "0771078635204f9c92c6d7affdcd6146"
format_stamp: "Formatted at 2026-06-20 11:45:16 on dist-test-slave-cjc6"
server_key: "51adefad54aaad11cbfeb7878e1de217"
server_key_iv: "bbe22f128939827c5b80d1b362603696"
server_key_version: "encryptionkey@0"
I20260620 11:45:37.240331 23288 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task1JRg7N/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781955746769051-7083-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-task1JRg7N/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781955746769051-7083-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-task1JRg7N/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781955746769051-7083-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 11:45:37.251863 23288 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 11:45:37.254165 23288 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 11:45:37.254338 23288 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 11:45:37.254580 23288 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 11:45:37.254881 23288 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 11:45:37.254935 23288 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 11:45:37.254974 23288 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 11:45:37.255003 23288 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 11:45:37.260730 23288 rpc_server.cc:307] RPC server started. Bound to: 127.6.234.195:34209
I20260620 11:45:37.260802 23416 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.6.234.195:34209 every 8 connection(s)
I20260620 11:45:37.261154 23288 server_base.cc:1166] Dumped server information to /tmp/dist-test-task1JRg7N/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781955746769051-7083-0/minicluster-data/ts-2/data/info.pb
Jun 20 11:45:37 dist-test-slave-cjc6 krb5kdc[22382](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781955937, etypes {rep=17 tkt=17 ses=17}, kudu/127.6.234.195@KRBTEST.COM for kudu/127.6.234.254@KRBTEST.COM
I20260620 11:45:37.267355 7083 external_mini_cluster.cc:1388] Started /tmp/dist-test-task1JRg7N/build/release/bin/kudu as pid 23288
I20260620 11:45:37.270978 23150 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 11:45:37.261992 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.6.234.195:56047 (local address 127.6.234.254:41033)
0620 11:45:37.262064 (+ 72us) server_negotiation.cc:206] Beginning negotiation
0620 11:45:37.262067 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0620 11:45:37.262260 (+ 193us) server_negotiation.cc:407] Connection header received
0620 11:45:37.262804 (+ 544us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 11:45:37.262806 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 11:45:37.262819 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 11:45:37.262872 (+ 53us) server_negotiation.cc:226] Negotiated authn=SASL
0620 11:45:37.263887 (+ 1015us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 11:45:37.264453 (+ 566us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 11:45:37.265456 (+ 1003us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 11:45:37.265671 (+ 215us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 11:45:37.265790 (+ 119us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 11:45:37.267976 (+ 2186us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 11:45:37.267977 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 11:45:37.267978 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 11:45:37.267991 (+ 13us) server_negotiation.cc:922] calling sasl_server_start()
0620 11:45:37.269575 (+ 1584us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 11:45:37.270058 (+ 483us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 11:45:37.270059 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 11:45:37.270059 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 11:45:37.270110 (+ 51us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 11:45:37.270366 (+ 256us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 11:45:37.270367 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 11:45:37.270367 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 11:45:37.270476 (+ 109us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 11:45:37.270560 (+ 84us) server_negotiation.cc:1035] Waiting for connection context
0620 11:45:37.270812 (+ 252us) server_negotiation.cc:299] Negotiation successful
0620 11:45:37.270898 (+ 86us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":43}
I20260620 11:45:37.271196 23419 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 11:45:37.261961 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.234.254:41033 (local address 127.6.234.195:56047)
0620 11:45:37.262120 (+ 159us) negotiation.cc:107] Waiting for socket to connect
0620 11:45:37.262161 (+ 41us) client_negotiation.cc:174] Beginning negotiation
0620 11:45:37.262651 (+ 490us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 11:45:37.262952 (+ 301us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 11:45:37.262956 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 11:45:37.263568 (+ 612us) client_negotiation.cc:189] Negotiated authn=SASL
0620 11:45:37.263765 (+ 197us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 11:45:37.263791 (+ 26us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 11:45:37.264554 (+ 763us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 11:45:37.264555 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 11:45:37.265364 (+ 809us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 11:45:37.265366 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 11:45:37.265804 (+ 438us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 11:45:37.265805 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 11:45:37.265929 (+ 124us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 11:45:37.266362 (+ 433us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 11:45:37.266371 (+ 9us) client_negotiation.cc:656] calling sasl_client_start()
0620 11:45:37.267837 (+ 1466us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 11:45:37.269644 (+ 1807us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 11:45:37.269650 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 11:45:37.269652 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0620 11:45:37.269977 (+ 325us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 11:45:37.270169 (+ 192us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 11:45:37.270170 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 11:45:37.270170 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0620 11:45:37.270306 (+ 136us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 11:45:37.270569 (+ 263us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 11:45:37.270574 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 11:45:37.270686 (+ 112us) client_negotiation.cc:769] Sending connection context
0620 11:45:37.270769 (+ 83us) client_negotiation.cc:240] Negotiation successful
0620 11:45:37.270943 (+ 174us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":117,"thread_start_us":87,"threads_started":1}
I20260620 11:45:37.271787 23417 heartbeater.cc:347] Connected to a master server at 127.6.234.254:41033
I20260620 11:45:37.271873 23417 heartbeater.cc:464] Registering TS with master...
I20260620 11:45:37.272053 23417 heartbeater.cc:511] Master 127.6.234.254:41033 requested a full tablet report, sending...
I20260620 11:45:37.272467 22959 ts_manager.cc:194] Registered new tserver with Master: 0771078635204f9c92c6d7affdcd6146 (127.6.234.195:34209)
I20260620 11:45:37.272971 22959 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.6.234.195@KRBTEST.COM'} at 127.6.234.195:56047
I20260620 11:45:37.275148 23420 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 11:45:37.267803 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:32794 (local address 127.6.234.254:41033)
0620 11:45:37.267927 (+ 124us) server_negotiation.cc:206] Beginning negotiation
0620 11:45:37.267929 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 11:45:37.268037 (+ 108us) server_negotiation.cc:407] Connection header received
0620 11:45:37.268064 (+ 27us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 11:45:37.268065 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 11:45:37.268076 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 11:45:37.268165 (+ 89us) server_negotiation.cc:226] Negotiated authn=SASL
0620 11:45:37.268706 (+ 541us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 11:45:37.269473 (+ 767us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 11:45:37.270523 (+ 1050us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 11:45:37.270778 (+ 255us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 11:45:37.270839 (+ 61us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 11:45:37.272506 (+ 1667us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 11:45:37.272507 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 11:45:37.272507 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 11:45:37.272515 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0620 11:45:37.274020 (+ 1505us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 11:45:37.274380 (+ 360us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 11:45:37.274381 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 11:45:37.274382 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 11:45:37.274438 (+ 56us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 11:45:37.274644 (+ 206us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 11:45:37.274645 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 11:45:37.274645 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 11:45:37.274753 (+ 108us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 11:45:37.274940 (+ 187us) server_negotiation.cc:1035] Waiting for connection context
0620 11:45:37.274977 (+ 37us) server_negotiation.cc:299] Negotiation successful
0620 11:45:37.275051 (+ 74us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":101,"thread_start_us":71,"threads_started":1}
I20260620 11:45:37.275761 7083 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260620 11:45:37.278169 23420 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 11:45:37.276065 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:32796 (local address 127.6.234.254:41033)
0620 11:45:37.276115 (+ 50us) server_negotiation.cc:206] Beginning negotiation
0620 11:45:37.276117 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 11:45:37.276215 (+ 98us) server_negotiation.cc:407] Connection header received
0620 11:45:37.276263 (+ 48us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 11:45:37.276264 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 11:45:37.276289 (+ 25us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 11:45:37.276356 (+ 67us) server_negotiation.cc:226] Negotiated authn=TOKEN
0620 11:45:37.276424 (+ 68us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 11:45:37.276852 (+ 428us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 11:45:37.277550 (+ 698us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 11:45:37.277782 (+ 232us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 11:45:37.277838 (+ 56us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 11:45:37.277970 (+ 132us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0620 11:45:37.278008 (+ 38us) server_negotiation.cc:394] Sending RPC error: FATAL_INVALID_AUTHENTICATION_TOKEN: Not authorized: token expired
0620 11:45:37.278129 (+ 121us) negotiation.cc:327] Negotiation complete: Not authorized: Server connection negotiation failed: server connection from 127.0.0.1:32796: token expired
Metrics: {"server-negotiator.queue_time_us":31}
W20260620 11:45:37.278208 23420 negotiation.cc:344] Unauthorized connection attempt: Server connection negotiation failed: server connection from 127.0.0.1:32796: token expired
W20260620 11:45:37.278288 22891 master_proxy_rpc.cc:203] Re-attempting CreateTable request to leader Master (127.6.234.254:41033)
I20260620 11:45:37.283571 23420 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 11:45:37.278567 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:32802 (local address 127.6.234.254:41033)
0620 11:45:37.278628 (+ 61us) server_negotiation.cc:206] Beginning negotiation
0620 11:45:37.278629 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0620 11:45:37.278637 (+ 8us) server_negotiation.cc:407] Connection header received
0620 11:45:37.278658 (+ 21us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 11:45:37.278659 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 11:45:37.278666 (+ 7us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 11:45:37.278705 (+ 39us) server_negotiation.cc:226] Negotiated authn=SASL
0620 11:45:37.279114 (+ 409us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 11:45:37.279537 (+ 423us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 11:45:37.280145 (+ 608us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 11:45:37.280306 (+ 161us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 11:45:37.280351 (+ 45us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 11:45:37.281286 (+ 935us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 11:45:37.281287 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 11:45:37.281288 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 11:45:37.281293 (+ 5us) server_negotiation.cc:922] calling sasl_server_start()
0620 11:45:37.282665 (+ 1372us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 11:45:37.282964 (+ 299us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 11:45:37.282965 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 11:45:37.282966 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 11:45:37.283018 (+ 52us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 11:45:37.283165 (+ 147us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 11:45:37.283165 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 11:45:37.283165 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 11:45:37.283260 (+ 95us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 11:45:37.283338 (+ 78us) server_negotiation.cc:1035] Waiting for connection context
0620 11:45:37.283411 (+ 73us) server_negotiation.cc:299] Negotiation successful
0620 11:45:37.283468 (+ 57us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":41}
I20260620 11:45:37.284713 22959 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:32802:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260620 11:45:37.286419 22959 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260620 11:45:37.288595 23081 tablet_service.cc:1467] Processing CreateTablet for tablet 19d3c59e62e54fb2a94fac22ee770963 (DEFAULT_TABLE table=test-table [id=5ae3f33efa8a48c688fd3e9123ac3160]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 11:45:37.288798 23081 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 19d3c59e62e54fb2a94fac22ee770963. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 11:45:37.290388 23039 tablet_bootstrap.cc:492] T 19d3c59e62e54fb2a94fac22ee770963 P 801a43964a204435806856f3de89b580: Bootstrap starting.
I20260620 11:45:37.290686 23039 tablet_bootstrap.cc:654] T 19d3c59e62e54fb2a94fac22ee770963 P 801a43964a204435806856f3de89b580: Neither blocks nor log segments found. Creating new log.
I20260620 11:45:37.291185 23039 tablet_bootstrap.cc:492] T 19d3c59e62e54fb2a94fac22ee770963 P 801a43964a204435806856f3de89b580: No bootstrap required, opened a new log
I20260620 11:45:37.291251 23039 ts_tablet_manager.cc:1397] T 19d3c59e62e54fb2a94fac22ee770963 P 801a43964a204435806856f3de89b580: Time spent bootstrapping tablet: real 0.001s user 0.000s sys 0.000s
I20260620 11:45:37.291608 23039 raft_consensus.cc:348] T 19d3c59e62e54fb2a94fac22ee770963 P 801a43964a204435806856f3de89b580 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0771078635204f9c92c6d7affdcd6146" member_type: VOTER last_known_addr { host: "127.6.234.195" port: 34209 } } peers { permanent_uuid: "13b3680a21f84e7e8f86c0d8aad45655" member_type: VOTER last_known_addr { host: "127.6.234.194" port: 38929 } } peers { permanent_uuid: "801a43964a204435806856f3de89b580" member_type: VOTER last_known_addr { host: "127.6.234.193" port: 41119 } }
I20260620 11:45:37.291694 23039 raft_consensus.cc:374] T 19d3c59e62e54fb2a94fac22ee770963 P 801a43964a204435806856f3de89b580 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 11:45:37.291741 23039 raft_consensus.cc:729] T 19d3c59e62e54fb2a94fac22ee770963 P 801a43964a204435806856f3de89b580 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 801a43964a204435806856f3de89b580, State: Initialized, Role: FOLLOWER
I20260620 11:45:37.291795 23039 consensus_queue.cc:260] T 19d3c59e62e54fb2a94fac22ee770963 P 801a43964a204435806856f3de89b580 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0771078635204f9c92c6d7affdcd6146" member_type: VOTER last_known_addr { host: "127.6.234.195" port: 34209 } } peers { permanent_uuid: "13b3680a21f84e7e8f86c0d8aad45655" member_type: VOTER last_known_addr { host: "127.6.234.194" port: 38929 } } peers { permanent_uuid: "801a43964a204435806856f3de89b580" member_type: VOTER last_known_addr { host: "127.6.234.193" port: 41119 } }
I20260620 11:45:37.291893 23039 ts_tablet_manager.cc:1428] T 19d3c59e62e54fb2a94fac22ee770963 P 801a43964a204435806856f3de89b580: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20260620 11:45:37.293678 23424 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 11:45:37.288345 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46478 (local address 127.6.234.195:34209)
0620 11:45:37.289891 (+ 1546us) server_negotiation.cc:206] Beginning negotiation
0620 11:45:37.289895 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 11:45:37.289904 (+ 9us) server_negotiation.cc:407] Connection header received
0620 11:45:37.289935 (+ 31us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 11:45:37.289937 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 11:45:37.289949 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 11:45:37.290057 (+ 108us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 11:45:37.290340 (+ 283us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 11:45:37.291485 (+ 1145us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 11:45:37.292717 (+ 1232us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 11:45:37.293297 (+ 580us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 11:45:37.293388 (+ 91us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 11:45:37.293447 (+ 59us) server_negotiation.cc:1035] Waiting for connection context
0620 11:45:37.293556 (+ 109us) server_negotiation.cc:299] Negotiation successful
0620 11:45:37.293593 (+ 37us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1513,"thread_start_us":123,"threads_started":1}
I20260620 11:45:37.293689 23154 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 11:45:37.288249 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.234.195:34209 (local address 127.0.0.1:46478)
0620 11:45:37.288315 (+ 66us) negotiation.cc:107] Waiting for socket to connect
0620 11:45:37.288326 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0620 11:45:37.288363 (+ 37us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 11:45:37.290064 (+ 1701us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 11:45:37.290065 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 11:45:37.290067 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 11:45:37.290127 (+ 60us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 11:45:37.290129 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 11:45:37.291599 (+ 1470us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 11:45:37.291600 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 11:45:37.292624 (+ 1024us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 11:45:37.292627 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 11:45:37.293379 (+ 752us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 11:45:37.293380 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 11:45:37.293462 (+ 82us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 11:45:37.293481 (+ 19us) client_negotiation.cc:769] Sending connection context
0620 11:45:37.293541 (+ 60us) client_negotiation.cc:240] Negotiation successful
0620 11:45:37.293583 (+ 42us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":49,"mutex_wait_us":33}
I20260620 11:45:37.293748 23425 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 11:45:37.289861 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41380 (local address 127.6.234.194:38929)
0620 11:45:37.290057 (+ 196us) server_negotiation.cc:206] Beginning negotiation
0620 11:45:37.290062 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 11:45:37.290073 (+ 11us) server_negotiation.cc:407] Connection header received
0620 11:45:37.290111 (+ 38us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 11:45:37.290112 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 11:45:37.290127 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 11:45:37.290201 (+ 74us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 11:45:37.290311 (+ 110us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 11:45:37.291496 (+ 1185us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 11:45:37.292751 (+ 1255us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 11:45:37.293327 (+ 576us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 11:45:37.293434 (+ 107us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 11:45:37.293490 (+ 56us) server_negotiation.cc:1035] Waiting for connection context
0620 11:45:37.293579 (+ 89us) server_negotiation.cc:299] Negotiation successful
0620 11:45:37.293616 (+ 37us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":163,"thread_start_us":105,"threads_started":1}
I20260620 11:45:37.293716 23423 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 11:45:37.288357 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.234.194:38929 (local address 127.0.0.1:41380)
0620 11:45:37.288520 (+ 163us) negotiation.cc:107] Waiting for socket to connect
0620 11:45:37.288529 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0620 11:45:37.288579 (+ 50us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 11:45:37.290202 (+ 1623us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 11:45:37.290203 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 11:45:37.290205 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 11:45:37.290251 (+ 46us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 11:45:37.290253 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 11:45:37.291574 (+ 1321us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 11:45:37.291575 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 11:45:37.292655 (+ 1080us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 11:45:37.292657 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 11:45:37.293440 (+ 783us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 11:45:37.293441 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 11:45:37.293504 (+ 63us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 11:45:37.293508 (+ 4us) client_negotiation.cc:769] Sending connection context
0620 11:45:37.293565 (+ 57us) client_negotiation.cc:240] Negotiation successful
0620 11:45:37.293606 (+ 41us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":147,"mutex_wait_us":103,"thread_start_us":119,"threads_started":1}
I20260620 11:45:37.294613 23351 tablet_service.cc:1467] Processing CreateTablet for tablet 19d3c59e62e54fb2a94fac22ee770963 (DEFAULT_TABLE table=test-table [id=5ae3f33efa8a48c688fd3e9123ac3160]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 11:45:37.294660 23219 tablet_service.cc:1467] Processing CreateTablet for tablet 19d3c59e62e54fb2a94fac22ee770963 (DEFAULT_TABLE table=test-table [id=5ae3f33efa8a48c688fd3e9123ac3160]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 11:45:37.294963 23351 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 19d3c59e62e54fb2a94fac22ee770963. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 11:45:37.294963 23219 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 19d3c59e62e54fb2a94fac22ee770963. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 11:45:37.298481 23427 tablet_bootstrap.cc:492] T 19d3c59e62e54fb2a94fac22ee770963 P 13b3680a21f84e7e8f86c0d8aad45655: Bootstrap starting.
I20260620 11:45:37.298576 23428 tablet_bootstrap.cc:492] T 19d3c59e62e54fb2a94fac22ee770963 P 0771078635204f9c92c6d7affdcd6146: Bootstrap starting.
I20260620 11:45:37.299206 23428 tablet_bootstrap.cc:654] T 19d3c59e62e54fb2a94fac22ee770963 P 0771078635204f9c92c6d7affdcd6146: Neither blocks nor log segments found. Creating new log.
I20260620 11:45:37.299206 23427 tablet_bootstrap.cc:654] T 19d3c59e62e54fb2a94fac22ee770963 P 13b3680a21f84e7e8f86c0d8aad45655: Neither blocks nor log segments found. Creating new log.
I20260620 11:45:37.299877 23427 log.cc:824] T 19d3c59e62e54fb2a94fac22ee770963 P 13b3680a21f84e7e8f86c0d8aad45655: Log is configured to *not* fsync() on all Append() calls
I20260620 11:45:37.299867 23428 log.cc:824] T 19d3c59e62e54fb2a94fac22ee770963 P 0771078635204f9c92c6d7affdcd6146: Log is configured to *not* fsync() on all Append() calls
I20260620 11:45:37.300729 23427 tablet_bootstrap.cc:492] T 19d3c59e62e54fb2a94fac22ee770963 P 13b3680a21f84e7e8f86c0d8aad45655: No bootstrap required, opened a new log
I20260620 11:45:37.300736 23428 tablet_bootstrap.cc:492] T 19d3c59e62e54fb2a94fac22ee770963 P 0771078635204f9c92c6d7affdcd6146: No bootstrap required, opened a new log
I20260620 11:45:37.300804 23427 ts_tablet_manager.cc:1397] T 19d3c59e62e54fb2a94fac22ee770963 P 13b3680a21f84e7e8f86c0d8aad45655: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20260620 11:45:37.300804 23428 ts_tablet_manager.cc:1397] T 19d3c59e62e54fb2a94fac22ee770963 P 0771078635204f9c92c6d7affdcd6146: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20260620 11:45:37.302743 23428 raft_consensus.cc:348] T 19d3c59e62e54fb2a94fac22ee770963 P 0771078635204f9c92c6d7affdcd6146 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0771078635204f9c92c6d7affdcd6146" member_type: VOTER last_known_addr { host: "127.6.234.195" port: 34209 } } peers { permanent_uuid: "13b3680a21f84e7e8f86c0d8aad45655" member_type: VOTER last_known_addr { host: "127.6.234.194" port: 38929 } } peers { permanent_uuid: "801a43964a204435806856f3de89b580" member_type: VOTER last_known_addr { host: "127.6.234.193" port: 41119 } }
I20260620 11:45:37.302743 23427 raft_consensus.cc:348] T 19d3c59e62e54fb2a94fac22ee770963 P 13b3680a21f84e7e8f86c0d8aad45655 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0771078635204f9c92c6d7affdcd6146" member_type: VOTER last_known_addr { host: "127.6.234.195" port: 34209 } } peers { permanent_uuid: "13b3680a21f84e7e8f86c0d8aad45655" member_type: VOTER last_known_addr { host: "127.6.234.194" port: 38929 } } peers { permanent_uuid: "801a43964a204435806856f3de89b580" member_type: VOTER last_known_addr { host: "127.6.234.193" port: 41119 } }
I20260620 11:45:37.303030 23428 raft_consensus.cc:374] T 19d3c59e62e54fb2a94fac22ee770963 P 0771078635204f9c92c6d7affdcd6146 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 11:45:37.303033 23427 raft_consensus.cc:374] T 19d3c59e62e54fb2a94fac22ee770963 P 13b3680a21f84e7e8f86c0d8aad45655 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 11:45:37.303081 23428 raft_consensus.cc:729] T 19d3c59e62e54fb2a94fac22ee770963 P 0771078635204f9c92c6d7affdcd6146 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0771078635204f9c92c6d7affdcd6146, State: Initialized, Role: FOLLOWER
I20260620 11:45:37.303081 23427 raft_consensus.cc:729] T 19d3c59e62e54fb2a94fac22ee770963 P 13b3680a21f84e7e8f86c0d8aad45655 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 13b3680a21f84e7e8f86c0d8aad45655, State: Initialized, Role: FOLLOWER
I20260620 11:45:37.303206 23428 consensus_queue.cc:260] T 19d3c59e62e54fb2a94fac22ee770963 P 0771078635204f9c92c6d7affdcd6146 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0771078635204f9c92c6d7affdcd6146" member_type: VOTER last_known_addr { host: "127.6.234.195" port: 34209 } } peers { permanent_uuid: "13b3680a21f84e7e8f86c0d8aad45655" member_type: VOTER last_known_addr { host: "127.6.234.194" port: 38929 } } peers { permanent_uuid: "801a43964a204435806856f3de89b580" member_type: VOTER last_known_addr { host: "127.6.234.193" port: 41119 } }
I20260620 11:45:37.303206 23427 consensus_queue.cc:260] T 19d3c59e62e54fb2a94fac22ee770963 P 13b3680a21f84e7e8f86c0d8aad45655 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0771078635204f9c92c6d7affdcd6146" member_type: VOTER last_known_addr { host: "127.6.234.195" port: 34209 } } peers { permanent_uuid: "13b3680a21f84e7e8f86c0d8aad45655" member_type: VOTER last_known_addr { host: "127.6.234.194" port: 38929 } } peers { permanent_uuid: "801a43964a204435806856f3de89b580" member_type: VOTER last_known_addr { host: "127.6.234.193" port: 41119 } }
I20260620 11:45:37.303498 23428 ts_tablet_manager.cc:1428] T 19d3c59e62e54fb2a94fac22ee770963 P 0771078635204f9c92c6d7affdcd6146: Time spent starting tablet: real 0.003s user 0.003s sys 0.000s
I20260620 11:45:37.303498 23427 ts_tablet_manager.cc:1428] T 19d3c59e62e54fb2a94fac22ee770963 P 13b3680a21f84e7e8f86c0d8aad45655: Time spent starting tablet: real 0.003s user 0.003s sys 0.000s
I20260620 11:45:37.303521 23285 heartbeater.cc:503] Master 127.6.234.254:41033 was elected leader, sending a full tablet report...
I20260620 11:45:37.303604 23417 heartbeater.cc:503] Master 127.6.234.254:41033 was elected leader, sending a full tablet report...
W20260620 11:45:37.382558 23286 tablet.cc:2369] T 19d3c59e62e54fb2a94fac22ee770963 P 13b3680a21f84e7e8f86c0d8aad45655: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260620 11:45:37.385437 23153 raft_consensus.cc:482] T 19d3c59e62e54fb2a94fac22ee770963 P 801a43964a204435806856f3de89b580 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260620 11:45:37.385540 23153 raft_consensus.cc:504] T 19d3c59e62e54fb2a94fac22ee770963 P 801a43964a204435806856f3de89b580 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0771078635204f9c92c6d7affdcd6146" member_type: VOTER last_known_addr { host: "127.6.234.195" port: 34209 } } peers { permanent_uuid: "13b3680a21f84e7e8f86c0d8aad45655" member_type: VOTER last_known_addr { host: "127.6.234.194" port: 38929 } } peers { permanent_uuid: "801a43964a204435806856f3de89b580" member_type: VOTER last_known_addr { host: "127.6.234.193" port: 41119 } }
I20260620 11:45:37.385878 23153 leader_election.cc:288] T 19d3c59e62e54fb2a94fac22ee770963 P 801a43964a204435806856f3de89b580 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 0771078635204f9c92c6d7affdcd6146 (127.6.234.195:34209), 13b3680a21f84e7e8f86c0d8aad45655 (127.6.234.194:38929)
I20260620 11:45:37.389823 23433 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 11:45:37.386168 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.234.194:38929 (local address 127.6.234.193:37585)
0620 11:45:37.386329 (+ 161us) negotiation.cc:107] Waiting for socket to connect
0620 11:45:37.386397 (+ 68us) client_negotiation.cc:174] Beginning negotiation
0620 11:45:37.386456 (+ 59us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 11:45:37.386605 (+ 149us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 11:45:37.386606 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 11:45:37.386608 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 11:45:37.386662 (+ 54us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 11:45:37.386663 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 11:45:37.387475 (+ 812us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 11:45:37.387475 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 11:45:37.388685 (+ 1210us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 11:45:37.388687 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 11:45:37.389445 (+ 758us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 11:45:37.389445 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 11:45:37.389534 (+ 89us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 11:45:37.389563 (+ 29us) client_negotiation.cc:769] Sending connection context
0620 11:45:37.389624 (+ 61us) client_negotiation.cc:240] Negotiation successful
0620 11:45:37.389666 (+ 42us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":145,"mutex_wait_us":25,"thread_start_us":109,"threads_started":1}
I20260620 11:45:37.389839 23425 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 11:45:37.386251 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.6.234.193:37585 (local address 127.6.234.194:38929)
0620 11:45:37.386298 (+ 47us) server_negotiation.cc:206] Beginning negotiation
0620 11:45:37.386300 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 11:45:37.386464 (+ 164us) server_negotiation.cc:407] Connection header received
0620 11:45:37.386514 (+ 50us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 11:45:37.386515 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 11:45:37.386523 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 11:45:37.386588 (+ 65us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 11:45:37.386722 (+ 134us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 11:45:37.387406 (+ 684us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 11:45:37.388775 (+ 1369us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 11:45:37.389349 (+ 574us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 11:45:37.389429 (+ 80us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 11:45:37.389438 (+ 9us) server_negotiation.cc:1035] Waiting for connection context
0620 11:45:37.389740 (+ 302us) server_negotiation.cc:299] Negotiation successful
0620 11:45:37.389780 (+ 40us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":21}
I20260620 11:45:37.389874 23149 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 11:45:37.386058 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.234.195:34209 (local address 127.6.234.193:60939)
0620 11:45:37.386125 (+ 67us) negotiation.cc:107] Waiting for socket to connect
0620 11:45:37.386139 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0620 11:45:37.386224 (+ 85us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 11:45:37.386428 (+ 204us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 11:45:37.386429 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 11:45:37.386430 (+ 1us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 11:45:37.386507 (+ 77us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 11:45:37.386509 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 11:45:37.387417 (+ 908us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 11:45:37.387418 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 11:45:37.388685 (+ 1267us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 11:45:37.388687 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 11:45:37.389443 (+ 756us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 11:45:37.389444 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 11:45:37.389534 (+ 90us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 11:45:37.389562 (+ 28us) client_negotiation.cc:769] Sending connection context
0620 11:45:37.389624 (+ 62us) client_negotiation.cc:240] Negotiation successful
0620 11:45:37.389666 (+ 42us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":39,"mutex_wait_us":44}
I20260620 11:45:37.390062 23424 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 11:45:37.386163 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.6.234.193:60939 (local address 127.6.234.195:34209)
0620 11:45:37.386218 (+ 55us) server_negotiation.cc:206] Beginning negotiation
0620 11:45:37.386220 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 11:45:37.386231 (+ 11us) server_negotiation.cc:407] Connection header received
0620 11:45:37.386340 (+ 109us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 11:45:37.386341 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 11:45:37.386348 (+ 7us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 11:45:37.386400 (+ 52us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 11:45:37.386580 (+ 180us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 11:45:37.387341 (+ 761us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 11:45:37.388775 (+ 1434us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 11:45:37.389349 (+ 574us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 11:45:37.389429 (+ 80us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 11:45:37.389438 (+ 9us) server_negotiation.cc:1035] Waiting for connection context
0620 11:45:37.389639 (+ 201us) server_negotiation.cc:299] Negotiation successful
0620 11:45:37.389676 (+ 37us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":29}
I20260620 11:45:37.390409 23239 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "19d3c59e62e54fb2a94fac22ee770963" candidate_uuid: "801a43964a204435806856f3de89b580" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "13b3680a21f84e7e8f86c0d8aad45655" is_pre_election: true
I20260620 11:45:37.390427 23371 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "19d3c59e62e54fb2a94fac22ee770963" candidate_uuid: "801a43964a204435806856f3de89b580" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0771078635204f9c92c6d7affdcd6146" is_pre_election: true
I20260620 11:45:37.390581 23239 raft_consensus.cc:2454] T 19d3c59e62e54fb2a94fac22ee770963 P 13b3680a21f84e7e8f86c0d8aad45655 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 801a43964a204435806856f3de89b580 in term 0.
I20260620 11:45:37.390583 23371 raft_consensus.cc:2454] T 19d3c59e62e54fb2a94fac22ee770963 P 0771078635204f9c92c6d7affdcd6146 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 801a43964a204435806856f3de89b580 in term 0.
I20260620 11:45:37.390790 23034 leader_election.cc:302] T 19d3c59e62e54fb2a94fac22ee770963 P 801a43964a204435806856f3de89b580 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0771078635204f9c92c6d7affdcd6146, 801a43964a204435806856f3de89b580; no voters:
I20260620 11:45:37.390929 23153 raft_consensus.cc:2781] T 19d3c59e62e54fb2a94fac22ee770963 P 801a43964a204435806856f3de89b580 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260620 11:45:37.391026 23153 raft_consensus.cc:482] T 19d3c59e62e54fb2a94fac22ee770963 P 801a43964a204435806856f3de89b580 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260620 11:45:37.391065 23153 raft_consensus.cc:3037] T 19d3c59e62e54fb2a94fac22ee770963 P 801a43964a204435806856f3de89b580 [term 0 FOLLOWER]: Advancing to term 1
I20260620 11:45:37.392057 23153 raft_consensus.cc:504] T 19d3c59e62e54fb2a94fac22ee770963 P 801a43964a204435806856f3de89b580 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0771078635204f9c92c6d7affdcd6146" member_type: VOTER last_known_addr { host: "127.6.234.195" port: 34209 } } peers { permanent_uuid: "13b3680a21f84e7e8f86c0d8aad45655" member_type: VOTER last_known_addr { host: "127.6.234.194" port: 38929 } } peers { permanent_uuid: "801a43964a204435806856f3de89b580" member_type: VOTER last_known_addr { host: "127.6.234.193" port: 41119 } }
I20260620 11:45:37.392215 23153 leader_election.cc:288] T 19d3c59e62e54fb2a94fac22ee770963 P 801a43964a204435806856f3de89b580 [CANDIDATE]: Term 1 election: Requested vote from peers 0771078635204f9c92c6d7affdcd6146 (127.6.234.195:34209), 13b3680a21f84e7e8f86c0d8aad45655 (127.6.234.194:38929)
I20260620 11:45:37.392408 23371 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "19d3c59e62e54fb2a94fac22ee770963" candidate_uuid: "801a43964a204435806856f3de89b580" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0771078635204f9c92c6d7affdcd6146"
I20260620 11:45:37.392494 23371 raft_consensus.cc:3037] T 19d3c59e62e54fb2a94fac22ee770963 P 0771078635204f9c92c6d7affdcd6146 [term 0 FOLLOWER]: Advancing to term 1
I20260620 11:45:37.392678 23239 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "19d3c59e62e54fb2a94fac22ee770963" candidate_uuid: "801a43964a204435806856f3de89b580" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "13b3680a21f84e7e8f86c0d8aad45655"
I20260620 11:45:37.392753 23239 raft_consensus.cc:3037] T 19d3c59e62e54fb2a94fac22ee770963 P 13b3680a21f84e7e8f86c0d8aad45655 [term 0 FOLLOWER]: Advancing to term 1
I20260620 11:45:37.393419 23371 raft_consensus.cc:2454] T 19d3c59e62e54fb2a94fac22ee770963 P 0771078635204f9c92c6d7affdcd6146 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 801a43964a204435806856f3de89b580 in term 1.
I20260620 11:45:37.393589 23034 leader_election.cc:302] T 19d3c59e62e54fb2a94fac22ee770963 P 801a43964a204435806856f3de89b580 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0771078635204f9c92c6d7affdcd6146, 801a43964a204435806856f3de89b580; no voters:
I20260620 11:45:37.393651 23239 raft_consensus.cc:2454] T 19d3c59e62e54fb2a94fac22ee770963 P 13b3680a21f84e7e8f86c0d8aad45655 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 801a43964a204435806856f3de89b580 in term 1.
I20260620 11:45:37.393721 23153 raft_consensus.cc:2781] T 19d3c59e62e54fb2a94fac22ee770963 P 801a43964a204435806856f3de89b580 [term 1 FOLLOWER]: Leader election won for term 1
I20260620 11:45:37.393887 23153 raft_consensus.cc:686] T 19d3c59e62e54fb2a94fac22ee770963 P 801a43964a204435806856f3de89b580 [term 1 LEADER]: Becoming Leader. State: Replica: 801a43964a204435806856f3de89b580, State: Running, Role: LEADER
I20260620 11:45:37.393970 23153 consensus_queue.cc:237] T 19d3c59e62e54fb2a94fac22ee770963 P 801a43964a204435806856f3de89b580 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0771078635204f9c92c6d7affdcd6146" member_type: VOTER last_known_addr { host: "127.6.234.195" port: 34209 } } peers { permanent_uuid: "13b3680a21f84e7e8f86c0d8aad45655" member_type: VOTER last_known_addr { host: "127.6.234.194" port: 38929 } } peers { permanent_uuid: "801a43964a204435806856f3de89b580" member_type: VOTER last_known_addr { host: "127.6.234.193" port: 41119 } }
I20260620 11:45:37.394837 22958 catalog_manager.cc:5495] T 19d3c59e62e54fb2a94fac22ee770963 P 801a43964a204435806856f3de89b580 reported cstate change: term changed from 0 to 1, leader changed from <none> to 801a43964a204435806856f3de89b580 (127.6.234.193). New cstate: current_term: 1 leader_uuid: "801a43964a204435806856f3de89b580" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0771078635204f9c92c6d7affdcd6146" member_type: VOTER last_known_addr { host: "127.6.234.195" port: 34209 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "13b3680a21f84e7e8f86c0d8aad45655" member_type: VOTER last_known_addr { host: "127.6.234.194" port: 38929 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "801a43964a204435806856f3de89b580" member_type: VOTER last_known_addr { host: "127.6.234.193" port: 41119 } health_report { overall_health: HEALTHY } } }
I20260620 11:45:37.408119 23155 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 11:45:37.405884 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60920 (local address 127.6.234.193:41119)
0620 11:45:37.405924 (+ 40us) server_negotiation.cc:206] Beginning negotiation
0620 11:45:37.405928 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 11:45:37.405945 (+ 17us) server_negotiation.cc:407] Connection header received
0620 11:45:37.406009 (+ 64us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 11:45:37.406009 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 11:45:37.406025 (+ 16us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 11:45:37.406081 (+ 56us) server_negotiation.cc:226] Negotiated authn=TOKEN
0620 11:45:37.406162 (+ 81us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 11:45:37.406626 (+ 464us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 11:45:37.407378 (+ 752us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 11:45:37.407570 (+ 192us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 11:45:37.407621 (+ 51us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 11:45:37.407824 (+ 203us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0620 11:45:37.407955 (+ 131us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0620 11:45:37.408009 (+ 54us) server_negotiation.cc:1035] Waiting for connection context
0620 11:45:37.408049 (+ 40us) server_negotiation.cc:299] Negotiation successful
0620 11:45:37.408069 (+ 20us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":17}
I20260620 11:45:37.409634 23371 raft_consensus.cc:1261] T 19d3c59e62e54fb2a94fac22ee770963 P 0771078635204f9c92c6d7affdcd6146 [term 1 FOLLOWER]: Refusing update from remote peer 801a43964a204435806856f3de89b580: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 11:45:37.409634 23239 raft_consensus.cc:1261] T 19d3c59e62e54fb2a94fac22ee770963 P 13b3680a21f84e7e8f86c0d8aad45655 [term 1 FOLLOWER]: Refusing update from remote peer 801a43964a204435806856f3de89b580: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 11:45:37.409977 23434 consensus_queue.cc:1035] T 19d3c59e62e54fb2a94fac22ee770963 P 801a43964a204435806856f3de89b580 [LEADER]: Connected to new peer: Peer: permanent_uuid: "13b3680a21f84e7e8f86c0d8aad45655" member_type: VOTER last_known_addr { host: "127.6.234.194" port: 38929 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 11:45:37.410109 23153 consensus_queue.cc:1035] T 19d3c59e62e54fb2a94fac22ee770963 P 801a43964a204435806856f3de89b580 [LEADER]: Connected to new peer: Peer: permanent_uuid: "0771078635204f9c92c6d7affdcd6146" member_type: VOTER last_known_addr { host: "127.6.234.195" port: 34209 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 11:45:37.412539 23436 mvcc.cc:205] Tried to move back new op lower bound from 7298891519627091968 to 7298891519566221312. Current Snapshot: MvccSnapshot[applied={T|T < 7298891519627091968}]
I20260620 11:45:37.413776 23438 mvcc.cc:205] Tried to move back new op lower bound from 7298891519627091968 to 7298891519566221312. Current Snapshot: MvccSnapshot[applied={T|T < 7298891519627091968}]
I20260620 11:45:37.414608 22958 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:32802:
table { table_name: "test-table" } modify_external_catalogs: true
I20260620 11:45:37.414708 22958 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:32802:
table { table_name: "test-table" } modify_external_catalogs: true
I20260620 11:45:37.415426 22958 catalog_manager.cc:5782] T 00000000000000000000000000000000 P d3dfdbcc949c49839481eebfb80e05d7: Sending DeleteTablet for 3 replicas of tablet 19d3c59e62e54fb2a94fac22ee770963
I20260620 11:45:37.415777 23351 tablet_service.cc:1514] Processing DeleteTablet for tablet 19d3c59e62e54fb2a94fac22ee770963 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 11:45:37 UTC) from {username='kudu', principal='kudu/127.6.234.254@KRBTEST.COM'} at 127.0.0.1:46478
I20260620 11:45:37.415778 23219 tablet_service.cc:1514] Processing DeleteTablet for tablet 19d3c59e62e54fb2a94fac22ee770963 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 11:45:37 UTC) from {username='kudu', principal='kudu/127.6.234.254@KRBTEST.COM'} at 127.0.0.1:41380
I20260620 11:45:37.416006 23446 tablet_replica.cc:330] T 19d3c59e62e54fb2a94fac22ee770963 P 0771078635204f9c92c6d7affdcd6146: stopping tablet replica
I20260620 11:45:37.416043 7083 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task1JRg7N/build/release/bin/kudu with pid 22997
I20260620 11:45:37.416066 23081 tablet_service.cc:1514] Processing DeleteTablet for tablet 19d3c59e62e54fb2a94fac22ee770963 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 11:45:37 UTC) from {username='kudu', principal='kudu/127.6.234.254@KRBTEST.COM'} at 127.0.0.1:60910
I20260620 11:45:37.416165 23446 raft_consensus.cc:2229] T 19d3c59e62e54fb2a94fac22ee770963 P 0771078635204f9c92c6d7affdcd6146 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260620 11:45:37.416283 23446 raft_consensus.cc:2258] T 19d3c59e62e54fb2a94fac22ee770963 P 0771078635204f9c92c6d7affdcd6146 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 11:45:37.416838 23445 tablet_replica.cc:330] T 19d3c59e62e54fb2a94fac22ee770963 P 13b3680a21f84e7e8f86c0d8aad45655: stopping tablet replica
I20260620 11:45:37.416936 23445 raft_consensus.cc:2229] T 19d3c59e62e54fb2a94fac22ee770963 P 13b3680a21f84e7e8f86c0d8aad45655 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260620 11:45:37.417035 23445 raft_consensus.cc:2258] T 19d3c59e62e54fb2a94fac22ee770963 P 13b3680a21f84e7e8f86c0d8aad45655 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 11:45:37.417466 23445 ts_tablet_manager.cc:1905] T 19d3c59e62e54fb2a94fac22ee770963 P 13b3680a21f84e7e8f86c0d8aad45655: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 11:45:37.417829 23446 ts_tablet_manager.cc:1905] T 19d3c59e62e54fb2a94fac22ee770963 P 0771078635204f9c92c6d7affdcd6146: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 11:45:37.419397 23446 ts_tablet_manager.cc:1918] T 19d3c59e62e54fb2a94fac22ee770963 P 0771078635204f9c92c6d7affdcd6146: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260620 11:45:37.419453 23445 ts_tablet_manager.cc:1918] T 19d3c59e62e54fb2a94fac22ee770963 P 13b3680a21f84e7e8f86c0d8aad45655: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260620 11:45:37.419483 23446 log.cc:1196] T 19d3c59e62e54fb2a94fac22ee770963 P 0771078635204f9c92c6d7affdcd6146: Deleting WAL directory at /tmp/dist-test-task1JRg7N/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781955746769051-7083-0/minicluster-data/ts-2/wal/wals/19d3c59e62e54fb2a94fac22ee770963
I20260620 11:45:37.419531 23445 log.cc:1196] T 19d3c59e62e54fb2a94fac22ee770963 P 13b3680a21f84e7e8f86c0d8aad45655: Deleting WAL directory at /tmp/dist-test-task1JRg7N/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781955746769051-7083-0/minicluster-data/ts-1/wal/wals/19d3c59e62e54fb2a94fac22ee770963
I20260620 11:45:37.419756 23446 ts_tablet_manager.cc:1939] T 19d3c59e62e54fb2a94fac22ee770963 P 0771078635204f9c92c6d7affdcd6146: Deleting consensus metadata
I20260620 11:45:37.419811 23445 ts_tablet_manager.cc:1939] T 19d3c59e62e54fb2a94fac22ee770963 P 13b3680a21f84e7e8f86c0d8aad45655: Deleting consensus metadata
I20260620 11:45:37.420166 22944 catalog_manager.cc:4841] TS 0771078635204f9c92c6d7affdcd6146 (127.6.234.195:34209): tablet 19d3c59e62e54fb2a94fac22ee770963 (table test-table [id=5ae3f33efa8a48c688fd3e9123ac3160]) successfully deleted
I20260620 11:45:37.420279 22944 catalog_manager.cc:4841] TS 13b3680a21f84e7e8f86c0d8aad45655 (127.6.234.194:38929): tablet 19d3c59e62e54fb2a94fac22ee770963 (table test-table [id=5ae3f33efa8a48c688fd3e9123ac3160]) successfully deleted
W20260620 11:45:37.422766 22943 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.6.234.193:41119 (error 108)
I20260620 11:45:37.423115 7083 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task1JRg7N/build/release/bin/kudu with pid 23152
I20260620 11:45:37.423251 23423 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 11:45:37.423103 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.234.193:41119 (local address 127.0.0.1:60930)
0620 11:45:37.423173 (+ 70us) negotiation.cc:107] Waiting for socket to connect
0620 11:45:37.423214 (+ 41us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.234.193:41119: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":47}
W20260620 11:45:37.425423 22943 catalog_manager.cc:4584] TS 801a43964a204435806856f3de89b580 (127.6.234.193:41119): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 19d3c59e62e54fb2a94fac22ee770963: Network error: Client connection negotiation failed: client connection to 127.6.234.193:41119: connect: Connection refused (error 111)
I20260620 11:45:37.429011 7083 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task1JRg7N/build/release/bin/kudu with pid 23288
I20260620 11:45:37.434329 7083 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task1JRg7N/build/release/bin/kudu with pid 22926
I20260620 11:45:57.439775 7083 external_mini_cluster.cc:1326] Running /tmp/dist-test-task1JRg7N/build/release/bin/kudu
/tmp/dist-test-task1JRg7N/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task1JRg7N/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781955746769051-7083-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task1JRg7N/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781955746769051-7083-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task1JRg7N/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781955746769051-7083-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task1JRg7N/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781955746769051-7083-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.6.234.254:41033
--webserver_interface=127.6.234.254
--webserver_port=42389
--keytab_file=/tmp/dist-test-task1JRg7N/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781955746769051-7083-0/krb5kdc/kudu_127.6.234.254.keytab
--principal=kudu/127.6.234.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.6.234.212:45931
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.6.234.254:41033
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--authn_token_validity_seconds=20
--authz_token_validity_seconds=20
--tsk_rotation_seconds=20 with env {KRB5CCNAME=/tmp/dist-test-task1JRg7N/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781955746769051-7083-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task1JRg7N/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781955746769051-7083-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task1JRg7N/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781955746769051-7083-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 11:45:57.521137 23448 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 11:45:57.521380 23448 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 11:45:57.521416 23448 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 11:45:57.522610 23448 flags.cc:425] Enabled experimental flag: --authn_token_validity_seconds=20
W20260620 11:45:57.522684 23448 flags.cc:425] Enabled experimental flag: --authz_token_validity_seconds=20
W20260620 11:45:57.522706 23448 flags.cc:425] Enabled experimental flag: --tsk_rotation_seconds=20
W20260620 11:45:57.522723 23448 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 11:45:57.522758 23448 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 11:45:57.522789 23448 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 11:45:57.522815 23448 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 11:45:57.522845 23448 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 11:45:57.524302 23448 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.6.234.212:45931
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task1JRg7N/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781955746769051-7083-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task1JRg7N/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781955746769051-7083-0/minicluster-data/master-0/wal
--authn_token_validity_seconds=20
--authz_token_validity_seconds=20
--tsk_rotation_seconds=20
--ipki_ca_key_size=768
--master_addresses=127.6.234.254:41033
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task1JRg7N/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781955746769051-7083-0/krb5kdc/kudu_127.6.234.254.keytab
--principal=kudu/127.6.234.254
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.6.234.254:41033
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task1JRg7N/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781955746769051-7083-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.6.234.254
--webserver_port=42389
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.23448
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task1JRg7N/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781955746769051-7083-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 20 Jun 2026 10:32:37 UTC on ccycloud.quasar-bgyofv.root.comops.site
build id 4457173
I20260620 11:45:57.524577 23448 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 11:45:57.524807 23448 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 11:45:57.529891 23454 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 11:45:57.529922 23453 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 11:45:57.529953 23448 server_base.cc:1034] running on GCE node
W20260620 11:45:57.529912 23456 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 11:45:57.530381 23448 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 11:45:57.530674 23448 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 11:45:57.531843 23448 hybrid_clock.cc:648] HybridClock initialized: now 1781955957531828 us; error 46 us; skew 500 ppm
Jun 20 11:45:57 dist-test-slave-cjc6 krb5kdc[22382](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781955957, etypes {rep=17 tkt=17 ses=17}, kudu/127.6.234.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 11:45:57.538776 23448 init.cc:377] Logged in from keytab as kudu/127.6.234.254@KRBTEST.COM (short username kudu)
I20260620 11:45:57.539204 23448 webserver.cc:460] Webserver started at http://127.6.234.254:42389/ using document root <none> and password file <none>
I20260620 11:45:57.539453 23448 fs_manager.cc:359] Metadata directory not provided
I20260620 11:45:57.539522 23448 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 11:45:57.540802 23448 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.001s sys 0.000s
I20260620 11:45:57.541450 23463 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 11:45:57.541617 23448 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260620 11:45:57.541733 23448 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task1JRg7N/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781955746769051-7083-0/minicluster-data/master-0/data,/tmp/dist-test-task1JRg7N/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781955746769051-7083-0/minicluster-data/master-0/wal
uuid: "d3dfdbcc949c49839481eebfb80e05d7"
format_stamp: "Formatted at 2026-06-20 11:45:16 on dist-test-slave-cjc6"
server_key: "d0eb388e2b925cb0e115a1ea9364db71"
server_key_iv: "03dce555a5f1af3eb04507723e819dc5"
server_key_version: "encryptionkey@0"
I20260620 11:45:57.542085 23448 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task1JRg7N/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781955746769051-7083-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task1JRg7N/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781955746769051-7083-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task1JRg7N/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781955746769051-7083-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 11:45:57.559188 23448 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 11:45:57.561122 23448 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 11:45:57.561298 23448 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 11:45:57.565376 23448 rpc_server.cc:307] RPC server started. Bound to: 127.6.234.254:41033
I20260620 11:45:57.565425 23515 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.6.234.254:41033 every 8 connection(s)
I20260620 11:45:57.565837 23448 server_base.cc:1166] Dumped server information to /tmp/dist-test-task1JRg7N/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781955746769051-7083-0/minicluster-data/master-0/data/info.pb
I20260620 11:45:57.567305 23516 sys_catalog.cc:263] Verifying existing consensus state
I20260620 11:45:57.567960 23516 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P d3dfdbcc949c49839481eebfb80e05d7: Bootstrap starting.
I20260620 11:45:57.575001 7083 external_mini_cluster.cc:1388] Started /tmp/dist-test-task1JRg7N/build/release/bin/kudu as pid 23448
I20260620 11:45:57.575387 7083 external_mini_cluster.cc:1326] Running /tmp/dist-test-task1JRg7N/build/release/bin/kudu
/tmp/dist-test-task1JRg7N/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task1JRg7N/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781955746769051-7083-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-task1JRg7N/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781955746769051-7083-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task1JRg7N/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781955746769051-7083-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-task1JRg7N/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781955746769051-7083-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.6.234.193:41119
--local_ip_for_outbound_sockets=127.6.234.193
--tserver_master_addrs=127.6.234.254:41033
--webserver_port=43611
--webserver_interface=127.6.234.193
--keytab_file=/tmp/dist-test-task1JRg7N/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781955746769051-7083-0/krb5kdc/kudu_127.6.234.193.keytab
--principal=kudu/127.6.234.193
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.6.234.212:45931
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task1JRg7N/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781955746769051-7083-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task1JRg7N/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781955746769051-7083-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task1JRg7N/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781955746769051-7083-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260620 11:45:57.577041 23516 log.cc:824] T 00000000000000000000000000000000 P d3dfdbcc949c49839481eebfb80e05d7: Log is configured to *not* fsync() on all Append() calls
I20260620 11:45:57.579488 23516 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P d3dfdbcc949c49839481eebfb80e05d7: Bootstrap replayed 1/1 log segments. Stats: ops{read=14 overwritten=0 applied=14 ignored=0} inserts{seen=8 ignored=0} mutations{seen=8 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260620 11:45:57.579794 23516 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P d3dfdbcc949c49839481eebfb80e05d7: Bootstrap complete.
I20260620 11:45:57.582059 23516 raft_consensus.cc:348] T 00000000000000000000000000000000 P d3dfdbcc949c49839481eebfb80e05d7 [term 2 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d3dfdbcc949c49839481eebfb80e05d7" member_type: VOTER last_known_addr { host: "127.6.234.254" port: 41033 } }
I20260620 11:45:57.582297 23516 raft_consensus.cc:729] T 00000000000000000000000000000000 P d3dfdbcc949c49839481eebfb80e05d7 [term 2 FOLLOWER]: Becoming Follower/Learner. State: Replica: d3dfdbcc949c49839481eebfb80e05d7, State: Initialized, Role: FOLLOWER
I20260620 11:45:57.582500 23516 consensus_queue.cc:260] T 00000000000000000000000000000000 P d3dfdbcc949c49839481eebfb80e05d7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 14, Last appended: 2.14, Last appended by leader: 14, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d3dfdbcc949c49839481eebfb80e05d7" member_type: VOTER last_known_addr { host: "127.6.234.254" port: 41033 } }
I20260620 11:45:57.582594 23516 raft_consensus.cc:388] T 00000000000000000000000000000000 P d3dfdbcc949c49839481eebfb80e05d7 [term 2 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260620 11:45:57.582648 23516 raft_consensus.cc:482] T 00000000000000000000000000000000 P d3dfdbcc949c49839481eebfb80e05d7 [term 2 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260620 11:45:57.582697 23516 raft_consensus.cc:3037] T 00000000000000000000000000000000 P d3dfdbcc949c49839481eebfb80e05d7 [term 2 FOLLOWER]: Advancing to term 3
I20260620 11:45:57.583663 23516 raft_consensus.cc:504] T 00000000000000000000000000000000 P d3dfdbcc949c49839481eebfb80e05d7 [term 3 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d3dfdbcc949c49839481eebfb80e05d7" member_type: VOTER last_known_addr { host: "127.6.234.254" port: 41033 } }
I20260620 11:45:57.583791 23516 leader_election.cc:302] T 00000000000000000000000000000000 P d3dfdbcc949c49839481eebfb80e05d7 [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: d3dfdbcc949c49839481eebfb80e05d7; no voters:
I20260620 11:45:57.583982 23516 leader_election.cc:288] T 00000000000000000000000000000000 P d3dfdbcc949c49839481eebfb80e05d7 [CANDIDATE]: Term 3 election: Requested vote from peers
I20260620 11:45:57.584023 23520 raft_consensus.cc:2781] T 00000000000000000000000000000000 P d3dfdbcc949c49839481eebfb80e05d7 [term 3 FOLLOWER]: Leader election won for term 3
I20260620 11:45:57.584137 23520 raft_consensus.cc:686] T 00000000000000000000000000000000 P d3dfdbcc949c49839481eebfb80e05d7 [term 3 LEADER]: Becoming Leader. State: Replica: d3dfdbcc949c49839481eebfb80e05d7, State: Running, Role: LEADER
I20260620 11:45:57.584254 23516 sys_catalog.cc:564] T 00000000000000000000000000000000 P d3dfdbcc949c49839481eebfb80e05d7 [sys.catalog]: configured and running, proceeding with master startup.
I20260620 11:45:57.584231 23520 consensus_queue.cc:237] T 00000000000000000000000000000000 P d3dfdbcc949c49839481eebfb80e05d7 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 14, Committed index: 14, Last appended: 2.14, Last appended by leader: 14, Current term: 3, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d3dfdbcc949c49839481eebfb80e05d7" member_type: VOTER last_known_addr { host: "127.6.234.254" port: 41033 } }
I20260620 11:45:57.584506 23520 sys_catalog.cc:455] T 00000000000000000000000000000000 P d3dfdbcc949c49839481eebfb80e05d7 [sys.catalog]: SysCatalogTable state changed. Reason: New leader d3dfdbcc949c49839481eebfb80e05d7. Latest consensus state: current_term: 3 leader_uuid: "d3dfdbcc949c49839481eebfb80e05d7" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d3dfdbcc949c49839481eebfb80e05d7" member_type: VOTER last_known_addr { host: "127.6.234.254" port: 41033 } } }
I20260620 11:45:57.584573 23520 sys_catalog.cc:458] T 00000000000000000000000000000000 P d3dfdbcc949c49839481eebfb80e05d7 [sys.catalog]: This master's current role is: LEADER
I20260620 11:45:57.584721 23521 sys_catalog.cc:455] T 00000000000000000000000000000000 P d3dfdbcc949c49839481eebfb80e05d7 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 3 leader_uuid: "d3dfdbcc949c49839481eebfb80e05d7" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d3dfdbcc949c49839481eebfb80e05d7" member_type: VOTER last_known_addr { host: "127.6.234.254" port: 41033 } } }
I20260620 11:45:57.584785 23524 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260620 11:45:57.584818 23521 sys_catalog.cc:458] T 00000000000000000000000000000000 P d3dfdbcc949c49839481eebfb80e05d7 [sys.catalog]: This master's current role is: LEADER
I20260620 11:45:57.585646 23524 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260620 11:45:57.585786 23524 catalog_manager.cc:1259] Loaded cluster ID: 5867141547be4785bce730c8f383fa7c
I20260620 11:45:57.585821 23524 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260620 11:45:57.586740 23524 catalog_manager.cc:1504] Loading token signing keys...
I20260620 11:45:57.586943 23524 catalog_manager.cc:5879] T 00000000000000000000000000000000 P d3dfdbcc949c49839481eebfb80e05d7: Loaded TSK: 0
I20260620 11:45:57.587001 23524 catalog_manager.cc:5879] T 00000000000000000000000000000000 P d3dfdbcc949c49839481eebfb80e05d7: Loaded TSK: 1
F20260620 11:45:57.587236 23524 token_signing_key.cc:66] Check failed: _s.ok() Bad status: Runtime error: unable to load data from memory: error:0D0680A8:asn1 encoding routines:ASN1_CHECK_TLEN:wrong tag:tasn_dec.c:1220 error:0D06C03A:asn1 encoding routines:ASN1_D2I_EX_PRIMITIVE:nested asn1 error:tasn_dec.c:788 error:0D08303A:asn1 encoding routines:ASN1_TEMPLATE_NOEXP_D2I:nested asn1 error:tasn_dec.c:720:Field=version, Type=RSA error:04093004:rsa routines:OLD_RSA_PRIV_DECODE:RSA lib:rsa_ameth.c:121 error:0D0680A8:asn1 encoding routines:ASN1_CHECK_TLEN:wrong tag:tasn_dec.c:1220 error:0D06C03A:asn1 encoding routines:ASN1_D2I_EX_PRIMITIVE:nested asn1 error:tasn_dec.c:788 error:0D08303A:asn1 encoding routines:ASN1_TEMPLATE_NOEXP_D2I:nested asn1 error:tasn_dec.c:720:Field=version, Type=PKCS8_PRIV_KEY_INFO
*** Check failure stack trace: ***
@ 0x7f6ba5c269ad google::LogMessage::Fail() at ??:0
@ 0x7f6ba5c2988a google::LogMessage::SendToLog() at ??:0
@ 0x7f6ba5c264de google::LogMessage::Flush() at ??:0
@ 0x7f6ba5c27a19 google::LogMessageFatal::~LogMessageFatal() at ??:0
@ 0x7f6ba8393104 kudu::security::TokenSigningPrivateKey::TokenSigningPrivateKey() at ??:0
@ 0x7f6ba8390d41 kudu::security::TokenSigner::ImportKeys() at ??:0
@ 0x7f6badb801cd kudu::master::CatalogManager::LoadTskEntries() at ??:0
@ 0x7f6badb8048e kudu::master::CatalogManager::InitTokenSigner() at ??:0
@ 0x7f6badb80594 _ZNSt17_Function_handlerIFN4kudu6StatusEvEZNS0_6master14CatalogManager24PrepareForLeadershipTaskEvEUlvE3_E9_M_invokeERKSt9_Any_data at ??:0
@ 0x7f6badb46c48 _ZZN4kudu6master14CatalogManager24PrepareForLeadershipTaskEvENKUlSt8functionIFNS_6StatusEvEERKNS_9consensus13RaftConsensusElPKcE_clES5_S9_lSB_.isra.591 at ??:0
@ 0x7f6badb50ad0 kudu::master::CatalogManager::PrepareForLeadershipTask() at ??:0
@ 0x7f6ba7af40d3 kudu::ThreadPool::DispatchThread() at ??:0
@ 0x7f6ba7ae9954 kudu::Thread::SuperviseThread() at ??:0
@ 0x7f6ba894d6db start_thread at ??:0
@ 0x7f6ba472461f clone at ??:0
W20260620 11:45:57.679838 23518 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 11:45:57.680037 23518 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 11:45:57.680060 23518 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 11:45:57.681221 23518 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 11:45:57.681263 23518 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 11:45:57.681300 23518 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.6.234.193
I20260620 11:45:57.682886 23518 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.6.234.212:45931
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task1JRg7N/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781955746769051-7083-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-task1JRg7N/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781955746769051-7083-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task1JRg7N/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781955746769051-7083-0/krb5kdc/kudu_127.6.234.193.keytab
--principal=kudu/127.6.234.193
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.6.234.193:41119
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task1JRg7N/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781955746769051-7083-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.6.234.193
--webserver_port=43611
--tserver_master_addrs=127.6.234.254:41033
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.23518
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.6.234.193
--log_dir=/tmp/dist-test-task1JRg7N/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781955746769051-7083-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 20 Jun 2026 10:32:37 UTC on ccycloud.quasar-bgyofv.root.comops.site
build id 4457173
I20260620 11:45:57.683152 23518 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 11:45:57.683384 23518 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 11:45:57.688886 23560 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 11:45:57.688911 23558 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 11:45:57.688947 23557 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 11:45:57.689116 23518 server_base.cc:1034] running on GCE node
I20260620 11:45:57.689308 23518 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 11:45:57.689492 23518 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 11:45:57.690618 23518 hybrid_clock.cc:648] HybridClock initialized: now 1781955957690587 us; error 50 us; skew 500 ppm
Jun 20 11:45:57 dist-test-slave-cjc6 krb5kdc[22382](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781955957, etypes {rep=17 tkt=17 ses=17}, kudu/127.6.234.193@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 11:45:57.693169 23518 init.cc:377] Logged in from keytab as kudu/127.6.234.193@KRBTEST.COM (short username kudu)
I20260620 11:45:57.693631 23518 webserver.cc:460] Webserver started at http://127.6.234.193:43611/ using document root <none> and password file <none>
I20260620 11:45:57.693888 23518 fs_manager.cc:359] Metadata directory not provided
I20260620 11:45:57.693981 23518 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 11:45:57.695492 23518 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.001s sys 0.000s
I20260620 11:45:57.696238 23567 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 11:45:57.696441 23518 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260620 11:45:57.696539 23518 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task1JRg7N/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781955746769051-7083-0/minicluster-data/ts-0/data,/tmp/dist-test-task1JRg7N/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781955746769051-7083-0/minicluster-data/ts-0/wal
uuid: "801a43964a204435806856f3de89b580"
format_stamp: "Formatted at 2026-06-20 11:45:16 on dist-test-slave-cjc6"
server_key: "cf24ad3d777c29baf2ef0bc5d01c92e7"
server_key_iv: "e03d6a10bd25f1620809f4643d5d419f"
server_key_version: "encryptionkey@0"
I20260620 11:45:57.696892 23518 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task1JRg7N/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781955746769051-7083-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-task1JRg7N/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781955746769051-7083-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-task1JRg7N/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781955746769051-7083-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 11:45:57.710598 23518 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 11:45:57.712999 23518 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 11:45:57.713158 23518 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 11:45:57.713428 23518 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 11:45:57.713951 23574 ts_tablet_manager.cc:536] Loading tablet metadata (0/1 complete)
I20260620 11:45:57.715845 23518 ts_tablet_manager.cc:579] Loaded tablet metadata (1 total tablets, 1 live tablets)
I20260620 11:45:57.715905 23518 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.002s user 0.000s sys 0.000s
I20260620 11:45:57.715950 23518 ts_tablet_manager.cc:594] Registering tablets (0/1 complete)
I20260620 11:45:57.716558 23518 ts_tablet_manager.cc:610] Registered 1 tablets
I20260620 11:45:57.716614 23518 ts_tablet_manager.cc:589] Time spent register tablets: real 0.001s user 0.000s sys 0.000s
I20260620 11:45:57.716677 23574 tablet_bootstrap.cc:492] T 19d3c59e62e54fb2a94fac22ee770963 P 801a43964a204435806856f3de89b580: Bootstrap starting.
I20260620 11:45:57.724112 23518 rpc_server.cc:307] RPC server started. Bound to: 127.6.234.193:41119
I20260620 11:45:57.724265 23681 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.6.234.193:41119 every 8 connection(s)
I20260620 11:45:57.724550 23518 server_base.cc:1166] Dumped server information to /tmp/dist-test-task1JRg7N/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781955746769051-7083-0/minicluster-data/ts-0/data/info.pb
I20260620 11:45:57.727540 23574 log.cc:824] T 19d3c59e62e54fb2a94fac22ee770963 P 801a43964a204435806856f3de89b580: Log is configured to *not* fsync() on all Append() calls
I20260620 11:45:57.730268 23574 tablet_bootstrap.cc:492] T 19d3c59e62e54fb2a94fac22ee770963 P 801a43964a204435806856f3de89b580: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=1 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260620 11:45:57.730585 23574 tablet_bootstrap.cc:492] T 19d3c59e62e54fb2a94fac22ee770963 P 801a43964a204435806856f3de89b580: Bootstrap complete.
I20260620 11:45:57.731117 7083 external_mini_cluster.cc:1388] Started /tmp/dist-test-task1JRg7N/build/release/bin/kudu as pid 23518
I20260620 11:45:57.731415 7083 external_mini_cluster.cc:1326] Running /tmp/dist-test-task1JRg7N/build/release/bin/kudu
/tmp/dist-test-task1JRg7N/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task1JRg7N/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781955746769051-7083-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-task1JRg7N/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781955746769051-7083-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task1JRg7N/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781955746769051-7083-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-task1JRg7N/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781955746769051-7083-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.6.234.194:38929
--local_ip_for_outbound_sockets=127.6.234.194
--tserver_master_addrs=127.6.234.254:41033
--webserver_port=33035
--webserver_interface=127.6.234.194
--keytab_file=/tmp/dist-test-task1JRg7N/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781955746769051-7083-0/krb5kdc/kudu_127.6.234.194.keytab
--principal=kudu/127.6.234.194
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.6.234.212:45931
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task1JRg7N/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781955746769051-7083-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task1JRg7N/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781955746769051-7083-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task1JRg7N/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781955746769051-7083-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260620 11:45:57.731756 23574 ts_tablet_manager.cc:1397] T 19d3c59e62e54fb2a94fac22ee770963 P 801a43964a204435806856f3de89b580: Time spent bootstrapping tablet: real 0.015s user 0.004s sys 0.009s
I20260620 11:45:57.733186 23574 raft_consensus.cc:348] T 19d3c59e62e54fb2a94fac22ee770963 P 801a43964a204435806856f3de89b580 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0771078635204f9c92c6d7affdcd6146" member_type: VOTER last_known_addr { host: "127.6.234.195" port: 34209 } } peers { permanent_uuid: "13b3680a21f84e7e8f86c0d8aad45655" member_type: VOTER last_known_addr { host: "127.6.234.194" port: 38929 } } peers { permanent_uuid: "801a43964a204435806856f3de89b580" member_type: VOTER last_known_addr { host: "127.6.234.193" port: 41119 } }
I20260620 11:45:57.733547 23574 raft_consensus.cc:729] T 19d3c59e62e54fb2a94fac22ee770963 P 801a43964a204435806856f3de89b580 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 801a43964a204435806856f3de89b580, State: Initialized, Role: FOLLOWER
I20260620 11:45:57.733750 23574 consensus_queue.cc:260] T 19d3c59e62e54fb2a94fac22ee770963 P 801a43964a204435806856f3de89b580 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0771078635204f9c92c6d7affdcd6146" member_type: VOTER last_known_addr { host: "127.6.234.195" port: 34209 } } peers { permanent_uuid: "13b3680a21f84e7e8f86c0d8aad45655" member_type: VOTER last_known_addr { host: "127.6.234.194" port: 38929 } } peers { permanent_uuid: "801a43964a204435806856f3de89b580" member_type: VOTER last_known_addr { host: "127.6.234.193" port: 41119 } }
I20260620 11:45:57.734047 23574 ts_tablet_manager.cc:1428] T 19d3c59e62e54fb2a94fac22ee770963 P 801a43964a204435806856f3de89b580: Time spent starting tablet: real 0.002s user 0.004s sys 0.000s
W20260620 11:45:57.778700 23684 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 11:45:57.725200 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.234.254:41033 (local address 127.6.234.193:46459)
0620 11:45:57.725373 (+ 173us) negotiation.cc:107] Waiting for socket to connect
0620 11:45:57.725417 (+ 44us) client_negotiation.cc:174] Beginning negotiation
0620 11:45:57.725992 (+ 575us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 11:45:57.778491 (+ 52499us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.234.254:41033: BlockingRecv error: recv error from unknown peer: Transport endpoint is not connected (error 107)
Metrics: {"client-negotiator.queue_time_us":131,"thread_start_us":104,"threads_started":1}
W20260620 11:45:57.778987 23682 heartbeater.cc:644] Failed to heartbeat to 127.6.234.254:41033 (0 consecutive failures): Network error: Failed to ping master at 127.6.234.254:41033: Client connection negotiation failed: client connection to 127.6.234.254:41033: BlockingRecv error: recv error from unknown peer: Transport endpoint is not connected (error 107)
I20260620 11:45:57.779323 23684 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 11:45:57.779206 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.234.254:41033 (local address 127.6.234.193:48431)
0620 11:45:57.779260 (+ 54us) negotiation.cc:107] Waiting for socket to connect
0620 11:45:57.779301 (+ 41us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.234.254:41033: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":36}
I20260620 11:45:57.779664 23684 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 11:45:57.779577 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.234.254:41033 (local address 127.6.234.193:47713)
0620 11:45:57.779620 (+ 43us) negotiation.cc:107] Waiting for socket to connect
0620 11:45:57.779646 (+ 26us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.234.254:41033: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":30}
W20260620 11:45:57.779774 23682 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
W20260620 11:45:57.812300 23686 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 11:45:57.812556 23686 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 11:45:57.812584 23686 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 11:45:57.813791 23686 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 11:45:57.813830 23686 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 11:45:57.813858 23686 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.6.234.194
I20260620 11:45:57.815220 23686 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.6.234.212:45931
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task1JRg7N/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781955746769051-7083-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-task1JRg7N/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781955746769051-7083-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task1JRg7N/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781955746769051-7083-0/krb5kdc/kudu_127.6.234.194.keytab
--principal=kudu/127.6.234.194
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.6.234.194:38929
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task1JRg7N/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781955746769051-7083-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.6.234.194
--webserver_port=33035
--tserver_master_addrs=127.6.234.254:41033
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.23686
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.6.234.194
--log_dir=/tmp/dist-test-task1JRg7N/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781955746769051-7083-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 20 Jun 2026 10:32:37 UTC on ccycloud.quasar-bgyofv.root.comops.site
build id 4457173
I20260620 11:45:57.815397 23686 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 11:45:57.815567 23686 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 11:45:57.820398 23692 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 11:45:57.820407 23695 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 11:45:57.820465 23693 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 11:45:57.820690 23686 server_base.cc:1034] running on GCE node
I20260620 11:45:57.820900 23686 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 11:45:57.821133 23686 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 11:45:57.822276 23686 hybrid_clock.cc:648] HybridClock initialized: now 1781955957822261 us; error 36 us; skew 500 ppm
Jun 20 11:45:57 dist-test-slave-cjc6 krb5kdc[22382](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781955957, etypes {rep=17 tkt=17 ses=17}, kudu/127.6.234.194@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 11:45:57.824771 23686 init.cc:377] Logged in from keytab as kudu/127.6.234.194@KRBTEST.COM (short username kudu)
I20260620 11:45:57.825232 23686 webserver.cc:460] Webserver started at http://127.6.234.194:33035/ using document root <none> and password file <none>
I20260620 11:45:57.825464 23686 fs_manager.cc:359] Metadata directory not provided
I20260620 11:45:57.825507 23686 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 11:45:57.826872 23686 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.000s sys 0.001s
I20260620 11:45:57.827461 23702 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 11:45:57.827632 23686 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.001s
I20260620 11:45:57.827728 23686 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task1JRg7N/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781955746769051-7083-0/minicluster-data/ts-1/data,/tmp/dist-test-task1JRg7N/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781955746769051-7083-0/minicluster-data/ts-1/wal
uuid: "13b3680a21f84e7e8f86c0d8aad45655"
format_stamp: "Formatted at 2026-06-20 11:45:16 on dist-test-slave-cjc6"
server_key: "8991967b56dcbd41188490e2e53c7cb0"
server_key_iv: "6896706017e3442902c05e3f611b85b0"
server_key_version: "encryptionkey@0"
I20260620 11:45:57.828065 23686 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task1JRg7N/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781955746769051-7083-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-task1JRg7N/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781955746769051-7083-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-task1JRg7N/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781955746769051-7083-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 11:45:57.836776 23686 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 11:45:57.839003 23686 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 11:45:57.839164 23686 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 11:45:57.839370 23686 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 11:45:57.839654 23686 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 11:45:57.839707 23686 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 11:45:57.839747 23686 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 11:45:57.839775 23686 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 11:45:57.845696 23686 rpc_server.cc:307] RPC server started. Bound to: 127.6.234.194:38929
I20260620 11:45:57.845769 23815 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.6.234.194:38929 every 8 connection(s)
I20260620 11:45:57.846076 23686 server_base.cc:1166] Dumped server information to /tmp/dist-test-task1JRg7N/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781955746769051-7083-0/minicluster-data/ts-1/data/info.pb
I20260620 11:45:57.846360 7083 external_mini_cluster.cc:1388] Started /tmp/dist-test-task1JRg7N/build/release/bin/kudu as pid 23686
I20260620 11:45:57.846695 7083 external_mini_cluster.cc:1326] Running /tmp/dist-test-task1JRg7N/build/release/bin/kudu
/tmp/dist-test-task1JRg7N/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task1JRg7N/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781955746769051-7083-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-task1JRg7N/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781955746769051-7083-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task1JRg7N/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781955746769051-7083-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-task1JRg7N/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781955746769051-7083-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.6.234.195:34209
--local_ip_for_outbound_sockets=127.6.234.195
--tserver_master_addrs=127.6.234.254:41033
--webserver_port=33353
--webserver_interface=127.6.234.195
--keytab_file=/tmp/dist-test-task1JRg7N/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781955746769051-7083-0/krb5kdc/kudu_127.6.234.195.keytab
--principal=kudu/127.6.234.195
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.6.234.212:45931
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task1JRg7N/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781955746769051-7083-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task1JRg7N/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781955746769051-7083-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task1JRg7N/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781955746769051-7083-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260620 11:45:57.847416 23818 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 11:45:57.847017 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.234.254:41033 (local address 127.6.234.194:43181)
0620 11:45:57.847180 (+ 163us) negotiation.cc:107] Waiting for socket to connect
0620 11:45:57.847254 (+ 74us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.234.254:41033: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":123,"thread_start_us":90,"threads_started":1}
W20260620 11:45:57.847910 23816 heartbeater.cc:644] Failed to heartbeat to 127.6.234.254:41033 (0 consecutive failures): Network error: Failed to ping master at 127.6.234.254:41033: Client connection negotiation failed: client connection to 127.6.234.254:41033: connect: Connection refused (error 111)
I20260620 11:45:57.848260 23818 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 11:45:57.848138 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.234.254:41033 (local address 127.6.234.194:56295)
0620 11:45:57.848197 (+ 59us) negotiation.cc:107] Waiting for socket to connect
0620 11:45:57.848232 (+ 35us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.234.254:41033: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":43}
I20260620 11:45:57.848529 23818 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 11:45:57.848443 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.234.254:41033 (local address 127.6.234.194:57953)
0620 11:45:57.848487 (+ 44us) negotiation.cc:107] Waiting for socket to connect
0620 11:45:57.848510 (+ 23us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.234.254:41033: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":30}
W20260620 11:45:57.848635 23816 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
W20260620 11:45:57.927546 23819 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 11:45:57.927762 23819 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 11:45:57.927785 23819 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 11:45:57.928859 23819 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 11:45:57.928894 23819 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 11:45:57.928920 23819 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.6.234.195
I20260620 11:45:57.930302 23819 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.6.234.212:45931
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task1JRg7N/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781955746769051-7083-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-task1JRg7N/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781955746769051-7083-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task1JRg7N/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781955746769051-7083-0/krb5kdc/kudu_127.6.234.195.keytab
--principal=kudu/127.6.234.195
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.6.234.195:34209
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task1JRg7N/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781955746769051-7083-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.6.234.195
--webserver_port=33353
--tserver_master_addrs=127.6.234.254:41033
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.23819
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.6.234.195
--log_dir=/tmp/dist-test-task1JRg7N/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781955746769051-7083-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 20 Jun 2026 10:32:37 UTC on ccycloud.quasar-bgyofv.root.comops.site
build id 4457173
I20260620 11:45:57.930536 23819 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 11:45:57.930807 23819 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 11:45:57.936062 23825 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 11:45:57.936062 23824 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 11:45:57.936112 23827 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 11:45:57.936316 23819 server_base.cc:1034] running on GCE node
I20260620 11:45:57.936515 23819 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 11:45:57.936722 23819 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 11:45:57.937844 23819 hybrid_clock.cc:648] HybridClock initialized: now 1781955957937814 us; error 44 us; skew 500 ppm
Jun 20 11:45:57 dist-test-slave-cjc6 krb5kdc[22382](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781955957, etypes {rep=17 tkt=17 ses=17}, kudu/127.6.234.195@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 11:45:57.940228 23819 init.cc:377] Logged in from keytab as kudu/127.6.234.195@KRBTEST.COM (short username kudu)
I20260620 11:45:57.940675 23819 webserver.cc:460] Webserver started at http://127.6.234.195:33353/ using document root <none> and password file <none>
I20260620 11:45:57.940914 23819 fs_manager.cc:359] Metadata directory not provided
I20260620 11:45:57.940958 23819 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 11:45:57.942358 23819 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.001s sys 0.000s
I20260620 11:45:57.942967 23834 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 11:45:57.943132 23819 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260620 11:45:57.943223 23819 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task1JRg7N/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781955746769051-7083-0/minicluster-data/ts-2/data,/tmp/dist-test-task1JRg7N/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781955746769051-7083-0/minicluster-data/ts-2/wal
uuid: "0771078635204f9c92c6d7affdcd6146"
format_stamp: "Formatted at 2026-06-20 11:45:16 on dist-test-slave-cjc6"
server_key: "51adefad54aaad11cbfeb7878e1de217"
server_key_iv: "bbe22f128939827c5b80d1b362603696"
server_key_version: "encryptionkey@0"
I20260620 11:45:57.943570 23819 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task1JRg7N/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781955746769051-7083-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-task1JRg7N/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781955746769051-7083-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-task1JRg7N/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781955746769051-7083-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 11:45:57.961889 23819 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 11:45:57.963712 23819 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 11:45:57.963881 23819 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 11:45:57.964118 23819 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 11:45:57.964435 23819 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 11:45:57.964491 23819 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 11:45:57.964526 23819 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 11:45:57.964553 23819 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 11:45:57.970414 23819 rpc_server.cc:307] RPC server started. Bound to: 127.6.234.195:34209
I20260620 11:45:57.970463 23947 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.6.234.195:34209 every 8 connection(s)
I20260620 11:45:57.970808 23819 server_base.cc:1166] Dumped server information to /tmp/dist-test-task1JRg7N/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781955746769051-7083-0/minicluster-data/ts-2/data/info.pb
I20260620 11:45:57.971655 7083 external_mini_cluster.cc:1388] Started /tmp/dist-test-task1JRg7N/build/release/bin/kudu as pid 23819
I20260620 11:45:57.971933 23950 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 11:45:57.971466 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.234.254:41033 (local address 127.6.234.195:48075)
0620 11:45:57.971635 (+ 169us) negotiation.cc:107] Waiting for socket to connect
0620 11:45:57.971715 (+ 80us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.234.254:41033: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":127,"thread_start_us":94,"threads_started":1}
W20260620 11:45:57.972420 23948 heartbeater.cc:644] Failed to heartbeat to 127.6.234.254:41033 (0 consecutive failures): Network error: Failed to ping master at 127.6.234.254:41033: Client connection negotiation failed: client connection to 127.6.234.254:41033: connect: Connection refused (error 111)
I20260620 11:45:57.972714 23950 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 11:45:57.972622 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.234.254:41033 (local address 127.6.234.195:56029)
0620 11:45:57.972673 (+ 51us) negotiation.cc:107] Waiting for socket to connect
0620 11:45:57.972698 (+ 25us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.234.254:41033: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":36}
I20260620 11:45:57.972975 23950 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 11:45:57.972901 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.234.254:41033 (local address 127.6.234.195:46679)
0620 11:45:57.972942 (+ 41us) negotiation.cc:107] Waiting for socket to connect
0620 11:45:57.972962 (+ 20us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.234.254:41033: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":30}
W20260620 11:45:57.973088 23948 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
I20260620 11:45:58.780525 23952 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 11:45:58.780231 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.234.254:41033 (local address 127.6.234.193:40959)
0620 11:45:58.780405 (+ 174us) negotiation.cc:107] Waiting for socket to connect
0620 11:45:58.780453 (+ 48us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.234.254:41033: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":151,"thread_start_us":73,"threads_started":1}
I20260620 11:45:58.849211 23953 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 11:45:58.848972 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.234.254:41033 (local address 127.6.234.194:56441)
0620 11:45:58.849119 (+ 147us) negotiation.cc:107] Waiting for socket to connect
0620 11:45:58.849158 (+ 39us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.234.254:41033: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":128,"thread_start_us":62,"threads_started":1}
I20260620 11:45:58.973560 23954 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 11:45:58.973375 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.234.254:41033 (local address 127.6.234.195:36299)
0620 11:45:58.973483 (+ 108us) negotiation.cc:107] Waiting for socket to connect
0620 11:45:58.973513 (+ 30us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.234.254:41033: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":95,"thread_start_us":59,"threads_started":1}
I20260620 11:45:59.162523 23955 raft_consensus.cc:482] T 19d3c59e62e54fb2a94fac22ee770963 P 801a43964a204435806856f3de89b580 [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260620 11:45:59.162669 23955 raft_consensus.cc:504] T 19d3c59e62e54fb2a94fac22ee770963 P 801a43964a204435806856f3de89b580 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0771078635204f9c92c6d7affdcd6146" member_type: VOTER last_known_addr { host: "127.6.234.195" port: 34209 } } peers { permanent_uuid: "13b3680a21f84e7e8f86c0d8aad45655" member_type: VOTER last_known_addr { host: "127.6.234.194" port: 38929 } } peers { permanent_uuid: "801a43964a204435806856f3de89b580" member_type: VOTER last_known_addr { host: "127.6.234.193" port: 41119 } }
I20260620 11:45:59.163049 23955 leader_election.cc:288] T 19d3c59e62e54fb2a94fac22ee770963 P 801a43964a204435806856f3de89b580 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 0771078635204f9c92c6d7affdcd6146 (127.6.234.195:34209), 13b3680a21f84e7e8f86c0d8aad45655 (127.6.234.194:38929)
Jun 20 11:45:59 dist-test-slave-cjc6 krb5kdc[22382](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781955957, etypes {rep=17 tkt=17 ses=17}, kudu/127.6.234.193@KRBTEST.COM for kudu/127.6.234.194@KRBTEST.COM
Jun 20 11:45:59 dist-test-slave-cjc6 krb5kdc[22382](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781955957, etypes {rep=17 tkt=17 ses=17}, kudu/127.6.234.193@KRBTEST.COM for kudu/127.6.234.195@KRBTEST.COM
I20260620 11:45:59.175717 23958 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 11:45:59.163492 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.6.234.193:51233 (local address 127.6.234.194:38929)
0620 11:45:59.163622 (+ 130us) server_negotiation.cc:206] Beginning negotiation
0620 11:45:59.163629 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0620 11:45:59.163657 (+ 28us) server_negotiation.cc:407] Connection header received
0620 11:45:59.164152 (+ 495us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 11:45:59.164155 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 11:45:59.164213 (+ 58us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 11:45:59.164368 (+ 155us) server_negotiation.cc:226] Negotiated authn=SASL
0620 11:45:59.165366 (+ 998us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 11:45:59.166791 (+ 1425us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 11:45:59.168386 (+ 1595us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 11:45:59.169023 (+ 637us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 11:45:59.169144 (+ 121us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 11:45:59.171964 (+ 2820us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 11:45:59.171968 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 11:45:59.171968 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 11:45:59.171978 (+ 10us) server_negotiation.cc:922] calling sasl_server_start()
0620 11:45:59.173995 (+ 2017us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 11:45:59.174503 (+ 508us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 11:45:59.174506 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 11:45:59.174507 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 11:45:59.174598 (+ 91us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 11:45:59.174852 (+ 254us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 11:45:59.174853 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 11:45:59.174853 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 11:45:59.175090 (+ 237us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 11:45:59.175182 (+ 92us) server_negotiation.cc:1035] Waiting for connection context
0620 11:45:59.175469 (+ 287us) server_negotiation.cc:299] Negotiation successful
0620 11:45:59.175588 (+ 119us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":92,"thread_start_us":58,"threads_started":1}
I20260620 11:45:59.175702 23956 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 11:45:59.163399 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.234.194:38929 (local address 127.6.234.193:51233)
0620 11:45:59.163513 (+ 114us) negotiation.cc:107] Waiting for socket to connect
0620 11:45:59.163525 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0620 11:45:59.163579 (+ 54us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 11:45:59.164453 (+ 874us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 11:45:59.164457 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 11:45:59.164961 (+ 504us) client_negotiation.cc:189] Negotiated authn=SASL
0620 11:45:59.165238 (+ 277us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 11:45:59.165245 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 11:45:59.166911 (+ 1666us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 11:45:59.166913 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 11:45:59.168223 (+ 1310us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 11:45:59.168226 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 11:45:59.169155 (+ 929us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 11:45:59.169156 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 11:45:59.169292 (+ 136us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 11:45:59.170011 (+ 719us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 11:45:59.170022 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0620 11:45:59.171863 (+ 1841us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 11:45:59.174105 (+ 2242us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 11:45:59.174112 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 11:45:59.174114 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0620 11:45:59.174406 (+ 292us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 11:45:59.174669 (+ 263us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 11:45:59.174670 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 11:45:59.174670 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0620 11:45:59.174783 (+ 113us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 11:45:59.175164 (+ 381us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 11:45:59.175172 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 11:45:59.175304 (+ 132us) client_negotiation.cc:769] Sending connection context
0620 11:45:59.175407 (+ 103us) client_negotiation.cc:240] Negotiation successful
0620 11:45:59.175600 (+ 193us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":101,"mutex_wait_us":64,"thread_start_us":53,"threads_started":1}
I20260620 11:45:59.175722 23957 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 11:45:59.163383 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.6.234.193:56991 (local address 127.6.234.195:34209)
0620 11:45:59.163529 (+ 146us) server_negotiation.cc:206] Beginning negotiation
0620 11:45:59.163540 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0620 11:45:59.163576 (+ 36us) server_negotiation.cc:407] Connection header received
0620 11:45:59.164153 (+ 577us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 11:45:59.164156 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 11:45:59.164213 (+ 57us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 11:45:59.164368 (+ 155us) server_negotiation.cc:226] Negotiated authn=SASL
0620 11:45:59.165366 (+ 998us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 11:45:59.166791 (+ 1425us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 11:45:59.168387 (+ 1596us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 11:45:59.169022 (+ 635us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 11:45:59.169144 (+ 122us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 11:45:59.172386 (+ 3242us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 11:45:59.172390 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 11:45:59.172390 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 11:45:59.172402 (+ 12us) server_negotiation.cc:922] calling sasl_server_start()
0620 11:45:59.174104 (+ 1702us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 11:45:59.174560 (+ 456us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 11:45:59.174563 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 11:45:59.174564 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 11:45:59.174649 (+ 85us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 11:45:59.174859 (+ 210us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 11:45:59.174860 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 11:45:59.174861 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 11:45:59.175084 (+ 223us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 11:45:59.175172 (+ 88us) server_negotiation.cc:1035] Waiting for connection context
0620 11:45:59.175471 (+ 299us) server_negotiation.cc:299] Negotiation successful
0620 11:45:59.175590 (+ 119us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":108,"thread_start_us":84,"threads_started":1}
I20260620 11:45:59.175887 23952 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 11:45:59.163263 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.234.195:34209 (local address 127.6.234.193:56991)
0620 11:45:59.163336 (+ 73us) negotiation.cc:107] Waiting for socket to connect
0620 11:45:59.163350 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0620 11:45:59.163417 (+ 67us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 11:45:59.164454 (+ 1037us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 11:45:59.164457 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 11:45:59.164961 (+ 504us) client_negotiation.cc:189] Negotiated authn=SASL
0620 11:45:59.165238 (+ 277us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 11:45:59.165245 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 11:45:59.166914 (+ 1669us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 11:45:59.166914 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 11:45:59.168223 (+ 1309us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 11:45:59.168226 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 11:45:59.169154 (+ 928us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 11:45:59.169155 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 11:45:59.169292 (+ 137us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 11:45:59.170010 (+ 718us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 11:45:59.170022 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0620 11:45:59.172245 (+ 2223us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 11:45:59.174197 (+ 1952us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 11:45:59.174198 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 11:45:59.174198 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0620 11:45:59.174468 (+ 270us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 11:45:59.174720 (+ 252us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 11:45:59.174721 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 11:45:59.174721 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0620 11:45:59.174794 (+ 73us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 11:45:59.175164 (+ 370us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 11:45:59.175170 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 11:45:59.175304 (+ 134us) client_negotiation.cc:769] Sending connection context
0620 11:45:59.175406 (+ 102us) client_negotiation.cc:240] Negotiation successful
0620 11:45:59.175683 (+ 277us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":35,"mutex_wait_us":233}
W20260620 11:45:59.176870 23569 leader_election.cc:341] T 19d3c59e62e54fb2a94fac22ee770963 P 801a43964a204435806856f3de89b580 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 0771078635204f9c92c6d7affdcd6146 (127.6.234.195:34209): Not found: Tablet not found: 19d3c59e62e54fb2a94fac22ee770963
W20260620 11:45:59.176967 23569 leader_election.cc:341] T 19d3c59e62e54fb2a94fac22ee770963 P 801a43964a204435806856f3de89b580 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 13b3680a21f84e7e8f86c0d8aad45655 (127.6.234.194:38929): Not found: Tablet not found: 19d3c59e62e54fb2a94fac22ee770963
I20260620 11:45:59.177019 23569 leader_election.cc:302] T 19d3c59e62e54fb2a94fac22ee770963 P 801a43964a204435806856f3de89b580 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 801a43964a204435806856f3de89b580; no voters: 0771078635204f9c92c6d7affdcd6146, 13b3680a21f84e7e8f86c0d8aad45655
I20260620 11:45:59.177170 23955 raft_consensus.cc:2726] T 19d3c59e62e54fb2a94fac22ee770963 P 801a43964a204435806856f3de89b580 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260620 11:45:59.781378 23959 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 11:45:59.781143 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.234.254:41033 (local address 127.6.234.193:46201)
0620 11:45:59.781290 (+ 147us) negotiation.cc:107] Waiting for socket to connect
0620 11:45:59.781329 (+ 39us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.234.254:41033: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":128,"thread_start_us":65,"threads_started":1}
I20260620 11:45:59.849890 23960 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 11:45:59.849666 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.234.254:41033 (local address 127.6.234.194:55681)
0620 11:45:59.849818 (+ 152us) negotiation.cc:107] Waiting for socket to connect
0620 11:45:59.849853 (+ 35us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.234.254:41033: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":136,"thread_start_us":72,"threads_started":1}
I20260620 11:45:59.974303 23961 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 11:45:59.974114 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.234.254:41033 (local address 127.6.234.195:53659)
0620 11:45:59.974229 (+ 115us) negotiation.cc:107] Waiting for socket to connect
0620 11:45:59.974266 (+ 37us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.234.254:41033: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":98,"thread_start_us":57,"threads_started":1}
I20260620 11:46:00.782169 23962 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 11:46:00.781919 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.234.254:41033 (local address 127.6.234.193:60503)
0620 11:46:00.782069 (+ 150us) negotiation.cc:107] Waiting for socket to connect
0620 11:46:00.782112 (+ 43us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.234.254:41033: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":127,"thread_start_us":69,"threads_started":1}
I20260620 11:46:00.807828 23963 raft_consensus.cc:482] T 19d3c59e62e54fb2a94fac22ee770963 P 801a43964a204435806856f3de89b580 [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260620 11:46:00.807921 23963 raft_consensus.cc:504] T 19d3c59e62e54fb2a94fac22ee770963 P 801a43964a204435806856f3de89b580 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0771078635204f9c92c6d7affdcd6146" member_type: VOTER last_known_addr { host: "127.6.234.195" port: 34209 } } peers { permanent_uuid: "13b3680a21f84e7e8f86c0d8aad45655" member_type: VOTER last_known_addr { host: "127.6.234.194" port: 38929 } } peers { permanent_uuid: "801a43964a204435806856f3de89b580" member_type: VOTER last_known_addr { host: "127.6.234.193" port: 41119 } }
I20260620 11:46:00.808089 23963 leader_election.cc:288] T 19d3c59e62e54fb2a94fac22ee770963 P 801a43964a204435806856f3de89b580 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 0771078635204f9c92c6d7affdcd6146 (127.6.234.195:34209), 13b3680a21f84e7e8f86c0d8aad45655 (127.6.234.194:38929)
W20260620 11:46:00.808646 23569 leader_election.cc:341] T 19d3c59e62e54fb2a94fac22ee770963 P 801a43964a204435806856f3de89b580 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 13b3680a21f84e7e8f86c0d8aad45655 (127.6.234.194:38929): Not found: Tablet not found: 19d3c59e62e54fb2a94fac22ee770963
W20260620 11:46:00.808738 23569 leader_election.cc:341] T 19d3c59e62e54fb2a94fac22ee770963 P 801a43964a204435806856f3de89b580 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 0771078635204f9c92c6d7affdcd6146 (127.6.234.195:34209): Not found: Tablet not found: 19d3c59e62e54fb2a94fac22ee770963
I20260620 11:46:00.808774 23569 leader_election.cc:302] T 19d3c59e62e54fb2a94fac22ee770963 P 801a43964a204435806856f3de89b580 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 801a43964a204435806856f3de89b580; no voters: 0771078635204f9c92c6d7affdcd6146, 13b3680a21f84e7e8f86c0d8aad45655
I20260620 11:46:00.808882 23963 raft_consensus.cc:2726] T 19d3c59e62e54fb2a94fac22ee770963 P 801a43964a204435806856f3de89b580 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260620 11:46:00.850544 23964 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 11:46:00.850313 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.234.254:41033 (local address 127.6.234.194:39769)
0620 11:46:00.850463 (+ 150us) negotiation.cc:107] Waiting for socket to connect
0620 11:46:00.850501 (+ 38us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.234.254:41033: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":133,"thread_start_us":61,"threads_started":1}
I20260620 11:46:00.975050 23965 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 11:46:00.974801 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.234.254:41033 (local address 127.6.234.195:44591)
0620 11:46:00.974959 (+ 158us) negotiation.cc:107] Waiting for socket to connect
0620 11:46:00.975004 (+ 45us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.234.254:41033: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":135,"thread_start_us":79,"threads_started":1}
I20260620 11:46:01.782876 23966 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 11:46:01.782644 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.234.254:41033 (local address 127.6.234.193:45183)
0620 11:46:01.782792 (+ 148us) negotiation.cc:107] Waiting for socket to connect
0620 11:46:01.782827 (+ 35us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.234.254:41033: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":132,"thread_start_us":89,"threads_started":1}
I20260620 11:46:01.851328 23967 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 11:46:01.851119 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.234.254:41033 (local address 127.6.234.194:46395)
0620 11:46:01.851248 (+ 129us) negotiation.cc:107] Waiting for socket to connect
0620 11:46:01.851289 (+ 41us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.234.254:41033: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":110,"thread_start_us":65,"threads_started":1}
I20260620 11:46:01.975803 23968 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 11:46:01.975593 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.234.254:41033 (local address 127.6.234.195:41561)
0620 11:46:01.975724 (+ 131us) negotiation.cc:107] Waiting for socket to connect
0620 11:46:01.975762 (+ 38us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.234.254:41033: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":114,"thread_start_us":75,"threads_started":1}
I20260620 11:46:02.783614 23969 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 11:46:02.783389 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.234.254:41033 (local address 127.6.234.193:47865)
0620 11:46:02.783524 (+ 135us) negotiation.cc:107] Waiting for socket to connect
0620 11:46:02.783564 (+ 40us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.234.254:41033: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":118,"thread_start_us":63,"threads_started":1}
I20260620 11:46:02.852046 23970 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 11:46:02.851839 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.234.254:41033 (local address 127.6.234.194:58935)
0620 11:46:02.851964 (+ 125us) negotiation.cc:107] Waiting for socket to connect
0620 11:46:02.852001 (+ 37us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.234.254:41033: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":107,"thread_start_us":58,"threads_started":1}
I20260620 11:46:02.976568 23971 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 11:46:02.976333 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.234.254:41033 (local address 127.6.234.195:43281)
0620 11:46:02.976492 (+ 159us) negotiation.cc:107] Waiting for socket to connect
0620 11:46:02.976530 (+ 38us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.234.254:41033: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":143,"thread_start_us":80,"threads_started":1}
I20260620 11:46:03.751248 23972 raft_consensus.cc:482] T 19d3c59e62e54fb2a94fac22ee770963 P 801a43964a204435806856f3de89b580 [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260620 11:46:03.751367 23972 raft_consensus.cc:504] T 19d3c59e62e54fb2a94fac22ee770963 P 801a43964a204435806856f3de89b580 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0771078635204f9c92c6d7affdcd6146" member_type: VOTER last_known_addr { host: "127.6.234.195" port: 34209 } } peers { permanent_uuid: "13b3680a21f84e7e8f86c0d8aad45655" member_type: VOTER last_known_addr { host: "127.6.234.194" port: 38929 } } peers { permanent_uuid: "801a43964a204435806856f3de89b580" member_type: VOTER last_known_addr { host: "127.6.234.193" port: 41119 } }
I20260620 11:46:03.751564 23972 leader_election.cc:288] T 19d3c59e62e54fb2a94fac22ee770963 P 801a43964a204435806856f3de89b580 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 0771078635204f9c92c6d7affdcd6146 (127.6.234.195:34209), 13b3680a21f84e7e8f86c0d8aad45655 (127.6.234.194:38929)
W20260620 11:46:03.751971 23569 leader_election.cc:341] T 19d3c59e62e54fb2a94fac22ee770963 P 801a43964a204435806856f3de89b580 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 0771078635204f9c92c6d7affdcd6146 (127.6.234.195:34209): Not found: Tablet not found: 19d3c59e62e54fb2a94fac22ee770963
W20260620 11:46:03.752053 23569 leader_election.cc:341] T 19d3c59e62e54fb2a94fac22ee770963 P 801a43964a204435806856f3de89b580 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 13b3680a21f84e7e8f86c0d8aad45655 (127.6.234.194:38929): Not found: Tablet not found: 19d3c59e62e54fb2a94fac22ee770963
I20260620 11:46:03.752069 23569 leader_election.cc:302] T 19d3c59e62e54fb2a94fac22ee770963 P 801a43964a204435806856f3de89b580 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 801a43964a204435806856f3de89b580; no voters: 0771078635204f9c92c6d7affdcd6146, 13b3680a21f84e7e8f86c0d8aad45655
I20260620 11:46:03.752143 23972 raft_consensus.cc:2726] T 19d3c59e62e54fb2a94fac22ee770963 P 801a43964a204435806856f3de89b580 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260620 11:46:03.784372 23973 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 11:46:03.784158 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.234.254:41033 (local address 127.6.234.193:60305)
0620 11:46:03.784286 (+ 128us) negotiation.cc:107] Waiting for socket to connect
0620 11:46:03.784324 (+ 38us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.234.254:41033: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":111,"thread_start_us":65,"threads_started":1}
I20260620 11:46:03.852839 23974 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 11:46:03.852590 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.234.254:41033 (local address 127.6.234.194:43801)
0620 11:46:03.852752 (+ 162us) negotiation.cc:107] Waiting for socket to connect
0620 11:46:03.852795 (+ 43us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.234.254:41033: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":141,"thread_start_us":80,"threads_started":1}
I20260620 11:46:03.977289 23975 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 11:46:03.977049 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.234.254:41033 (local address 127.6.234.195:46311)
0620 11:46:03.977205 (+ 156us) negotiation.cc:107] Waiting for socket to connect
0620 11:46:03.977246 (+ 41us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.234.254:41033: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":137,"thread_start_us":78,"threads_started":1}
I20260620 11:46:04.785091 23976 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 11:46:04.784858 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.234.254:41033 (local address 127.6.234.193:44013)
0620 11:46:04.785006 (+ 148us) negotiation.cc:107] Waiting for socket to connect
0620 11:46:04.785044 (+ 38us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.234.254:41033: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":128,"thread_start_us":62,"threads_started":1}
I20260620 11:46:04.853639 23977 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 11:46:04.853407 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.234.254:41033 (local address 127.6.234.194:35919)
0620 11:46:04.853554 (+ 147us) negotiation.cc:107] Waiting for socket to connect
0620 11:46:04.853594 (+ 40us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.234.254:41033: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":129,"thread_start_us":62,"threads_started":1}
I20260620 11:46:04.978169 23978 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 11:46:04.977903 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.234.254:41033 (local address 127.6.234.195:59423)
0620 11:46:04.978082 (+ 179us) negotiation.cc:107] Waiting for socket to connect
0620 11:46:04.978126 (+ 44us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.234.254:41033: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":159,"thread_start_us":122,"threads_started":1}
I20260620 11:46:05.786015 23979 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 11:46:05.785753 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.234.254:41033 (local address 127.6.234.193:44951)
0620 11:46:05.785917 (+ 164us) negotiation.cc:107] Waiting for socket to connect
0620 11:46:05.785965 (+ 48us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.234.254:41033: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":141,"thread_start_us":80,"threads_started":1}
I20260620 11:46:05.854396 23980 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 11:46:05.854177 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.234.254:41033 (local address 127.6.234.194:49615)
0620 11:46:05.854315 (+ 138us) negotiation.cc:107] Waiting for socket to connect
0620 11:46:05.854352 (+ 37us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.234.254:41033: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":121,"thread_start_us":76,"threads_started":1}
I20260620 11:46:05.978873 23981 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 11:46:05.978623 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.234.254:41033 (local address 127.6.234.195:55963)
0620 11:46:05.978787 (+ 164us) negotiation.cc:107] Waiting for socket to connect
0620 11:46:05.978827 (+ 40us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.234.254:41033: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":145,"thread_start_us":77,"threads_started":1}
I20260620 11:46:06.786816 23982 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 11:46:06.786563 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.234.254:41033 (local address 127.6.234.193:56881)
0620 11:46:06.786720 (+ 157us) negotiation.cc:107] Waiting for socket to connect
0620 11:46:06.786767 (+ 47us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.234.254:41033: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":138,"thread_start_us":83,"threads_started":1}
I20260620 11:46:06.855084 23983 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 11:46:06.854844 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.234.254:41033 (local address 127.6.234.194:33921)
0620 11:46:06.854996 (+ 152us) negotiation.cc:107] Waiting for socket to connect
0620 11:46:06.855039 (+ 43us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.234.254:41033: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":133,"thread_start_us":63,"threads_started":1}
I20260620 11:46:06.979604 23984 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 11:46:06.979397 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.234.254:41033 (local address 127.6.234.195:43875)
0620 11:46:06.979514 (+ 117us) negotiation.cc:107] Waiting for socket to connect
0620 11:46:06.979552 (+ 38us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.234.254:41033: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":101,"thread_start_us":62,"threads_started":1}
I20260620 11:46:07.787621 23985 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 11:46:07.787383 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.234.254:41033 (local address 127.6.234.193:37201)
0620 11:46:07.787531 (+ 148us) negotiation.cc:107] Waiting for socket to connect
0620 11:46:07.787571 (+ 40us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.234.254:41033: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":130,"thread_start_us":69,"threads_started":1}
I20260620 11:46:07.855772 23986 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 11:46:07.855565 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.234.254:41033 (local address 127.6.234.194:50963)
0620 11:46:07.855693 (+ 128us) negotiation.cc:107] Waiting for socket to connect
0620 11:46:07.855729 (+ 36us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.234.254:41033: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":111,"thread_start_us":61,"threads_started":1}
I20260620 11:46:07.980405 23987 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 11:46:07.980115 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.234.254:41033 (local address 127.6.234.195:59625)
0620 11:46:07.980295 (+ 180us) negotiation.cc:107] Waiting for socket to connect
0620 11:46:07.980343 (+ 48us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.234.254:41033: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":157,"thread_start_us":60,"threads_started":1}
I20260620 11:46:08.232401 23988 raft_consensus.cc:482] T 19d3c59e62e54fb2a94fac22ee770963 P 801a43964a204435806856f3de89b580 [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260620 11:46:08.232539 23988 raft_consensus.cc:504] T 19d3c59e62e54fb2a94fac22ee770963 P 801a43964a204435806856f3de89b580 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0771078635204f9c92c6d7affdcd6146" member_type: VOTER last_known_addr { host: "127.6.234.195" port: 34209 } } peers { permanent_uuid: "13b3680a21f84e7e8f86c0d8aad45655" member_type: VOTER last_known_addr { host: "127.6.234.194" port: 38929 } } peers { permanent_uuid: "801a43964a204435806856f3de89b580" member_type: VOTER last_known_addr { host: "127.6.234.193" port: 41119 } }
I20260620 11:46:08.232750 23988 leader_election.cc:288] T 19d3c59e62e54fb2a94fac22ee770963 P 801a43964a204435806856f3de89b580 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 0771078635204f9c92c6d7affdcd6146 (127.6.234.195:34209), 13b3680a21f84e7e8f86c0d8aad45655 (127.6.234.194:38929)
W20260620 11:46:08.233162 23569 leader_election.cc:341] T 19d3c59e62e54fb2a94fac22ee770963 P 801a43964a204435806856f3de89b580 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 0771078635204f9c92c6d7affdcd6146 (127.6.234.195:34209): Not found: Tablet not found: 19d3c59e62e54fb2a94fac22ee770963
W20260620 11:46:08.233244 23569 leader_election.cc:341] T 19d3c59e62e54fb2a94fac22ee770963 P 801a43964a204435806856f3de89b580 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 13b3680a21f84e7e8f86c0d8aad45655 (127.6.234.194:38929): Not found: Tablet not found: 19d3c59e62e54fb2a94fac22ee770963
I20260620 11:46:08.233261 23569 leader_election.cc:302] T 19d3c59e62e54fb2a94fac22ee770963 P 801a43964a204435806856f3de89b580 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 801a43964a204435806856f3de89b580; no voters: 0771078635204f9c92c6d7affdcd6146, 13b3680a21f84e7e8f86c0d8aad45655
I20260620 11:46:08.233331 23988 raft_consensus.cc:2726] T 19d3c59e62e54fb2a94fac22ee770963 P 801a43964a204435806856f3de89b580 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260620 11:46:08.788414 23989 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 11:46:08.788135 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.234.254:41033 (local address 127.6.234.193:53155)
0620 11:46:08.788322 (+ 187us) negotiation.cc:107] Waiting for socket to connect
0620 11:46:08.788364 (+ 42us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.234.254:41033: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":167,"thread_start_us":99,"threads_started":1}
I20260620 11:46:08.856442 23990 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 11:46:08.856210 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.234.254:41033 (local address 127.6.234.194:54601)
0620 11:46:08.856346 (+ 136us) negotiation.cc:107] Waiting for socket to connect
0620 11:46:08.856378 (+ 32us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.234.254:41033: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":119,"thread_start_us":78,"threads_started":1}
I20260620 11:46:08.981191 23991 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 11:46:08.980963 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.234.254:41033 (local address 127.6.234.195:54543)
0620 11:46:08.981096 (+ 133us) negotiation.cc:107] Waiting for socket to connect
0620 11:46:08.981132 (+ 36us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.234.254:41033: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":117,"thread_start_us":77,"threads_started":1}
I20260620 11:46:09.789150 23992 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 11:46:09.788895 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.234.254:41033 (local address 127.6.234.193:59109)
0620 11:46:09.789052 (+ 157us) negotiation.cc:107] Waiting for socket to connect
0620 11:46:09.789094 (+ 42us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.234.254:41033: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":138,"thread_start_us":80,"threads_started":1}
I20260620 11:46:09.857143 23993 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 11:46:09.856873 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.234.254:41033 (local address 127.6.234.194:49263)
0620 11:46:09.857057 (+ 184us) negotiation.cc:107] Waiting for socket to connect
0620 11:46:09.857094 (+ 37us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.234.254:41033: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":168,"thread_start_us":112,"threads_started":1}
I20260620 11:46:09.982026 23994 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 11:46:09.981800 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.234.254:41033 (local address 127.6.234.195:39547)
0620 11:46:09.981946 (+ 146us) negotiation.cc:107] Waiting for socket to connect
0620 11:46:09.981981 (+ 35us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.234.254:41033: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":128,"thread_start_us":84,"threads_started":1}
I20260620 11:46:10.790047 23995 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 11:46:10.789744 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.234.254:41033 (local address 127.6.234.193:47553)
0620 11:46:10.789893 (+ 149us) negotiation.cc:107] Waiting for socket to connect
0620 11:46:10.789936 (+ 43us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.234.254:41033: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":127,"thread_start_us":77,"threads_started":1}
I20260620 11:46:10.857831 23996 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 11:46:10.857614 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.234.254:41033 (local address 127.6.234.194:44565)
0620 11:46:10.857747 (+ 133us) negotiation.cc:107] Waiting for socket to connect
0620 11:46:10.857785 (+ 38us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.234.254:41033: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":116,"thread_start_us":56,"threads_started":1}
I20260620 11:46:10.982693 23997 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 11:46:10.982464 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.234.254:41033 (local address 127.6.234.195:37267)
0620 11:46:10.982610 (+ 146us) negotiation.cc:107] Waiting for socket to connect
0620 11:46:10.982647 (+ 37us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.234.254:41033: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":120,"thread_start_us":81,"threads_started":1}
I20260620 11:46:11.790767 23998 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 11:46:11.790514 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.234.254:41033 (local address 127.6.234.193:55683)
0620 11:46:11.790676 (+ 162us) negotiation.cc:107] Waiting for socket to connect
0620 11:46:11.790711 (+ 35us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.234.254:41033: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":139,"thread_start_us":106,"threads_started":1}
I20260620 11:46:11.858515 23999 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 11:46:11.858298 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.234.254:41033 (local address 127.6.234.194:41503)
0620 11:46:11.858436 (+ 138us) negotiation.cc:107] Waiting for socket to connect
0620 11:46:11.858470 (+ 34us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.234.254:41033: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":122,"thread_start_us":78,"threads_started":1}
I20260620 11:46:11.983440 24000 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 11:46:11.983226 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.234.254:41033 (local address 127.6.234.195:34295)
0620 11:46:11.983347 (+ 121us) negotiation.cc:107] Waiting for socket to connect
0620 11:46:11.983385 (+ 38us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.234.254:41033: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":105,"thread_start_us":62,"threads_started":1}
I20260620 11:46:12.506387 24001 raft_consensus.cc:482] T 19d3c59e62e54fb2a94fac22ee770963 P 801a43964a204435806856f3de89b580 [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260620 11:46:12.506479 24001 raft_consensus.cc:504] T 19d3c59e62e54fb2a94fac22ee770963 P 801a43964a204435806856f3de89b580 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0771078635204f9c92c6d7affdcd6146" member_type: VOTER last_known_addr { host: "127.6.234.195" port: 34209 } } peers { permanent_uuid: "13b3680a21f84e7e8f86c0d8aad45655" member_type: VOTER last_known_addr { host: "127.6.234.194" port: 38929 } } peers { permanent_uuid: "801a43964a204435806856f3de89b580" member_type: VOTER last_known_addr { host: "127.6.234.193" port: 41119 } }
I20260620 11:46:12.506650 24001 leader_election.cc:288] T 19d3c59e62e54fb2a94fac22ee770963 P 801a43964a204435806856f3de89b580 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 0771078635204f9c92c6d7affdcd6146 (127.6.234.195:34209), 13b3680a21f84e7e8f86c0d8aad45655 (127.6.234.194:38929)
W20260620 11:46:12.507063 23569 leader_election.cc:341] T 19d3c59e62e54fb2a94fac22ee770963 P 801a43964a204435806856f3de89b580 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 0771078635204f9c92c6d7affdcd6146 (127.6.234.195:34209): Not found: Tablet not found: 19d3c59e62e54fb2a94fac22ee770963
W20260620 11:46:12.507144 23569 leader_election.cc:341] T 19d3c59e62e54fb2a94fac22ee770963 P 801a43964a204435806856f3de89b580 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 13b3680a21f84e7e8f86c0d8aad45655 (127.6.234.194:38929): Not found: Tablet not found: 19d3c59e62e54fb2a94fac22ee770963
I20260620 11:46:12.507176 23569 leader_election.cc:302] T 19d3c59e62e54fb2a94fac22ee770963 P 801a43964a204435806856f3de89b580 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 801a43964a204435806856f3de89b580; no voters: 0771078635204f9c92c6d7affdcd6146, 13b3680a21f84e7e8f86c0d8aad45655
I20260620 11:46:12.507268 24001 raft_consensus.cc:2726] T 19d3c59e62e54fb2a94fac22ee770963 P 801a43964a204435806856f3de89b580 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260620 11:46:12.791473 24002 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 11:46:12.791235 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.234.254:41033 (local address 127.6.234.193:58777)
0620 11:46:12.791369 (+ 134us) negotiation.cc:107] Waiting for socket to connect
0620 11:46:12.791413 (+ 44us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.234.254:41033: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":114,"thread_start_us":79,"threads_started":1}
I20260620 11:46:12.859181 24003 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 11:46:12.858993 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.234.254:41033 (local address 127.6.234.194:39401)
0620 11:46:12.859107 (+ 114us) negotiation.cc:107] Waiting for socket to connect
0620 11:46:12.859141 (+ 34us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.234.254:41033: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":98,"thread_start_us":55,"threads_started":1}
I20260620 11:46:12.978776 7083 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task1JRg7N/build/release/bin/kudu with pid 23518
I20260620 11:46:12.983733 7083 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task1JRg7N/build/release/bin/kudu with pid 23686
I20260620 11:46:12.985446 24004 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 11:46:12.984970 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.234.254:41033 (local address 127.6.234.195:53031)
0620 11:46:12.985372 (+ 402us) negotiation.cc:107] Waiting for socket to connect
0620 11:46:12.985406 (+ 34us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.234.254:41033: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":373,"thread_start_us":55,"threads_started":1}
I20260620 11:46:12.988338 7083 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task1JRg7N/build/release/bin/kudu with pid 23819
2026-06-20T11:46:12Z chronyd exiting
[ OK ] SecurityITest.TestEnableAndDisableEncryptedTSK (58195 ms)
[----------] 30 tests from SecurityITest (226227 ms total)
[----------] 6 tests from EncryptionPolicyTest
[ RUN ] EncryptionPolicyTest.TestEncryptionPolicy/0
Loading random data
Initializing database '/tmp/dist-test-task1JRg7N/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781955746769051-7083-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 20 11:46:13 dist-test-slave-cjc6 krb5kdc[24007](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 20 11:46:13 dist-test-slave-cjc6 krb5kdc[24007](info): set up 2 sockets
Jun 20 11:46:13 dist-test-slave-cjc6 krb5kdc[24007](info): commencing operation
krb5kdc: starting...
W20260620 11:46:14.025393 7083 mini_kdc.cc:121] Time spent starting KDC: real 1.017s user 0.000s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task1JRg7N/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781955746769051-7083-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task1JRg7N/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781955746769051-7083-0/krb5kdc/test-user.keytab.
Jun 20 11:46:14 dist-test-slave-cjc6 krb5kdc[24007](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781955974, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-20T11:46:14Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-20T11:46:14Z Disabled control of system clock
WARNING: no policy specified for kudu/127.6.234.254@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.6.234.254@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.6.234.254 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task1JRg7N/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781955746769051-7083-0/krb5kdc/kudu_127.6.234.254.keytab.
Entry for principal kudu/127.6.234.254 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task1JRg7N/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781955746769051-7083-0/krb5kdc/kudu_127.6.234.254.keytab.
I20260620 11:46:14.142647 7083 external_mini_cluster.cc:1326] Running /tmp/dist-test-task1JRg7N/build/release/bin/kudu
/tmp/dist-test-task1JRg7N/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task1JRg7N/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781955746769051-7083-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task1JRg7N/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781955746769051-7083-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task1JRg7N/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781955746769051-7083-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task1JRg7N/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781955746769051-7083-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.6.234.254:38591
--webserver_interface=127.6.234.254
--webserver_port=0
--keytab_file=/tmp/dist-test-task1JRg7N/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781955746769051-7083-0/krb5kdc/kudu_127.6.234.254.keytab
--principal=kudu/127.6.234.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.6.234.212:37185
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.6.234.254:38591
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task1JRg7N/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781955746769051-7083-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task1JRg7N/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781955746769051-7083-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task1JRg7N/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781955746769051-7083-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 11:46:14.223738 24020 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 11:46:14.223963 24020 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 11:46:14.224009 24020 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 11:46:14.225121 24020 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 11:46:14.225188 24020 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 11:46:14.225211 24020 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 11:46:14.225232 24020 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 11:46:14.225262 24020 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 11:46:14.226693 24020 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.6.234.212:37185
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task1JRg7N/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781955746769051-7083-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task1JRg7N/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781955746769051-7083-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.6.234.254:38591
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task1JRg7N/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781955746769051-7083-0/krb5kdc/kudu_127.6.234.254.keytab
--principal=kudu/127.6.234.254
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.6.234.254:38591
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task1JRg7N/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781955746769051-7083-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.6.234.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.24020
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task1JRg7N/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781955746769051-7083-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 20 Jun 2026 10:32:37 UTC on ccycloud.quasar-bgyofv.root.comops.site
build id 4457173
I20260620 11:46:14.226961 24020 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 11:46:14.227181 24020 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 11:46:14.232419 24025 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 11:46:14.232492 24028 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 11:46:14.232535 24020 server_base.cc:1034] running on GCE node
W20260620 11:46:14.232488 24026 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 11:46:14.232892 24020 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 11:46:14.233130 24020 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 11:46:14.234267 24020 hybrid_clock.cc:648] HybridClock initialized: now 1781955974234248 us; error 36 us; skew 500 ppm
Jun 20 11:46:14 dist-test-slave-cjc6 krb5kdc[24007](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781955974, etypes {rep=17 tkt=17 ses=17}, kudu/127.6.234.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 11:46:14.236749 24020 init.cc:377] Logged in from keytab as kudu/127.6.234.254@KRBTEST.COM (short username kudu)
I20260620 11:46:14.237180 24020 webserver.cc:460] Webserver started at http://127.6.234.254:46809/ using document root <none> and password file <none>
I20260620 11:46:14.237401 24020 fs_manager.cc:359] Metadata directory not provided
I20260620 11:46:14.237468 24020 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 11:46:14.237622 24020 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 11:46:14.238690 24020 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task1JRg7N/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781955746769051-7083-0/minicluster-data/master-0/data/instance:
uuid: "36daf303b10940278e2db95c7b2a1398"
format_stamp: "Formatted at 2026-06-20 11:46:14 on dist-test-slave-cjc6"
server_key: "5ea445a7af8d328acdd485ba7fbd5132"
server_key_iv: "a893a90d5f440f6d563e966b046e43da"
server_key_version: "encryptionkey@0"
I20260620 11:46:14.239048 24020 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task1JRg7N/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781955746769051-7083-0/minicluster-data/master-0/wal/instance:
uuid: "36daf303b10940278e2db95c7b2a1398"
format_stamp: "Formatted at 2026-06-20 11:46:14 on dist-test-slave-cjc6"
server_key: "5ea445a7af8d328acdd485ba7fbd5132"
server_key_iv: "a893a90d5f440f6d563e966b046e43da"
server_key_version: "encryptionkey@0"
I20260620 11:46:14.240310 24020 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260620 11:46:14.240994 24035 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 11:46:14.241163 24020 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260620 11:46:14.241259 24020 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task1JRg7N/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781955746769051-7083-0/minicluster-data/master-0/data,/tmp/dist-test-task1JRg7N/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781955746769051-7083-0/minicluster-data/master-0/wal
uuid: "36daf303b10940278e2db95c7b2a1398"
format_stamp: "Formatted at 2026-06-20 11:46:14 on dist-test-slave-cjc6"
server_key: "5ea445a7af8d328acdd485ba7fbd5132"
server_key_iv: "a893a90d5f440f6d563e966b046e43da"
server_key_version: "encryptionkey@0"
I20260620 11:46:14.241340 24020 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task1JRg7N/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781955746769051-7083-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task1JRg7N/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781955746769051-7083-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task1JRg7N/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781955746769051-7083-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 11:46:14.256639 24020 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 11:46:14.258589 24020 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 11:46:14.258749 24020 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 11:46:14.262452 24020 rpc_server.cc:307] RPC server started. Bound to: 127.6.234.254:38591
I20260620 11:46:14.262487 24087 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.6.234.254:38591 every 8 connection(s)
I20260620 11:46:14.262825 24020 server_base.cc:1166] Dumped server information to /tmp/dist-test-task1JRg7N/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781955746769051-7083-0/minicluster-data/master-0/data/info.pb
I20260620 11:46:14.263643 24088 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 11:46:14.265908 24088 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 36daf303b10940278e2db95c7b2a1398: Bootstrap starting.
I20260620 11:46:14.266484 24088 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 36daf303b10940278e2db95c7b2a1398: Neither blocks nor log segments found. Creating new log.
I20260620 11:46:14.266764 24088 log.cc:824] T 00000000000000000000000000000000 P 36daf303b10940278e2db95c7b2a1398: Log is configured to *not* fsync() on all Append() calls
I20260620 11:46:14.267107 7083 external_mini_cluster.cc:1388] Started /tmp/dist-test-task1JRg7N/build/release/bin/kudu as pid 24020
I20260620 11:46:14.267225 7083 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task1JRg7N/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781955746769051-7083-0/minicluster-data/master-0/wal/instance
I20260620 11:46:14.267417 7083 external_mini_cluster.cc:1428] Setting key 748e6f8d85a718a0e7feaf9055977b18
I20260620 11:46:14.267577 24088 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 36daf303b10940278e2db95c7b2a1398: No bootstrap required, opened a new log
I20260620 11:46:14.269402 24088 raft_consensus.cc:348] T 00000000000000000000000000000000 P 36daf303b10940278e2db95c7b2a1398 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "36daf303b10940278e2db95c7b2a1398" member_type: VOTER last_known_addr { host: "127.6.234.254" port: 38591 } }
I20260620 11:46:14.269569 24088 raft_consensus.cc:374] T 00000000000000000000000000000000 P 36daf303b10940278e2db95c7b2a1398 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 11:46:14.269616 24088 raft_consensus.cc:729] T 00000000000000000000000000000000 P 36daf303b10940278e2db95c7b2a1398 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 36daf303b10940278e2db95c7b2a1398, State: Initialized, Role: FOLLOWER
I20260620 11:46:14.269761 24088 consensus_queue.cc:260] T 00000000000000000000000000000000 P 36daf303b10940278e2db95c7b2a1398 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "36daf303b10940278e2db95c7b2a1398" member_type: VOTER last_known_addr { host: "127.6.234.254" port: 38591 } }
I20260620 11:46:14.269855 24088 raft_consensus.cc:388] T 00000000000000000000000000000000 P 36daf303b10940278e2db95c7b2a1398 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260620 11:46:14.269898 24088 raft_consensus.cc:482] T 00000000000000000000000000000000 P 36daf303b10940278e2db95c7b2a1398 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260620 11:46:14.269945 24088 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 36daf303b10940278e2db95c7b2a1398 [term 0 FOLLOWER]: Advancing to term 1
I20260620 11:46:14.270452 24088 raft_consensus.cc:504] T 00000000000000000000000000000000 P 36daf303b10940278e2db95c7b2a1398 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "36daf303b10940278e2db95c7b2a1398" member_type: VOTER last_known_addr { host: "127.6.234.254" port: 38591 } }
I20260620 11:46:14.270568 24088 leader_election.cc:302] T 00000000000000000000000000000000 P 36daf303b10940278e2db95c7b2a1398 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 36daf303b10940278e2db95c7b2a1398; no voters:
I20260620 11:46:14.270735 24088 leader_election.cc:288] T 00000000000000000000000000000000 P 36daf303b10940278e2db95c7b2a1398 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260620 11:46:14.270757 24093 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 36daf303b10940278e2db95c7b2a1398 [term 1 FOLLOWER]: Leader election won for term 1
I20260620 11:46:14.270924 24093 raft_consensus.cc:686] T 00000000000000000000000000000000 P 36daf303b10940278e2db95c7b2a1398 [term 1 LEADER]: Becoming Leader. State: Replica: 36daf303b10940278e2db95c7b2a1398, State: Running, Role: LEADER
I20260620 11:46:14.270972 24088 sys_catalog.cc:564] T 00000000000000000000000000000000 P 36daf303b10940278e2db95c7b2a1398 [sys.catalog]: configured and running, proceeding with master startup.
I20260620 11:46:14.271054 24093 consensus_queue.cc:237] T 00000000000000000000000000000000 P 36daf303b10940278e2db95c7b2a1398 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "36daf303b10940278e2db95c7b2a1398" member_type: VOTER last_known_addr { host: "127.6.234.254" port: 38591 } }
I20260620 11:46:14.271461 24093 sys_catalog.cc:455] T 00000000000000000000000000000000 P 36daf303b10940278e2db95c7b2a1398 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 36daf303b10940278e2db95c7b2a1398. Latest consensus state: current_term: 1 leader_uuid: "36daf303b10940278e2db95c7b2a1398" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "36daf303b10940278e2db95c7b2a1398" member_type: VOTER last_known_addr { host: "127.6.234.254" port: 38591 } } }
I20260620 11:46:14.271539 24093 sys_catalog.cc:458] T 00000000000000000000000000000000 P 36daf303b10940278e2db95c7b2a1398 [sys.catalog]: This master's current role is: LEADER
I20260620 11:46:14.271793 24094 sys_catalog.cc:455] T 00000000000000000000000000000000 P 36daf303b10940278e2db95c7b2a1398 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "36daf303b10940278e2db95c7b2a1398" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "36daf303b10940278e2db95c7b2a1398" member_type: VOTER last_known_addr { host: "127.6.234.254" port: 38591 } } }
I20260620 11:46:14.271919 24099 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260620 11:46:14.271929 24094 sys_catalog.cc:458] T 00000000000000000000000000000000 P 36daf303b10940278e2db95c7b2a1398 [sys.catalog]: This master's current role is: LEADER
I20260620 11:46:14.272475 24099 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260620 11:46:14.274256 24099 catalog_manager.cc:1347] Generated new cluster ID: 62dbd38eb66845c39b4ac6cac90e5c81
I20260620 11:46:14.274315 24099 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
Jun 20 11:46:14 dist-test-slave-cjc6 krb5kdc[24007](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781955974, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.6.234.254@KRBTEST.COM
I20260620 11:46:14.279368 24092 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 11:46:14.267948 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37892 (local address 127.6.234.254:38591)
0620 11:46:14.268137 (+ 189us) server_negotiation.cc:206] Beginning negotiation
0620 11:46:14.268145 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0620 11:46:14.268180 (+ 35us) server_negotiation.cc:407] Connection header received
0620 11:46:14.268754 (+ 574us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 11:46:14.268760 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 11:46:14.268804 (+ 44us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 11:46:14.268951 (+ 147us) server_negotiation.cc:226] Negotiated authn=SASL
0620 11:46:14.269550 (+ 599us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 11:46:14.271050 (+ 1500us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 11:46:14.272159 (+ 1109us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 11:46:14.272824 (+ 665us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 11:46:14.272945 (+ 121us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 11:46:14.275826 (+ 2881us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 11:46:14.275831 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 11:46:14.275831 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 11:46:14.275842 (+ 11us) server_negotiation.cc:922] calling sasl_server_start()
0620 11:46:14.277777 (+ 1935us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 11:46:14.278180 (+ 403us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 11:46:14.278182 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 11:46:14.278183 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 11:46:14.278282 (+ 99us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 11:46:14.278512 (+ 230us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 11:46:14.278513 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 11:46:14.278514 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 11:46:14.278793 (+ 279us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 11:46:14.278932 (+ 139us) server_negotiation.cc:1035] Waiting for connection context
0620 11:46:14.279057 (+ 125us) server_negotiation.cc:299] Negotiation successful
0620 11:46:14.279182 (+ 125us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":130,"thread_start_us":80,"threads_started":1}
I20260620 11:46:14.289075 24099 catalog_manager.cc:1370] Generated new certificate authority record
I20260620 11:46:14.289608 24099 catalog_manager.cc:1504] Loading token signing keys...
I20260620 11:46:14.293275 24099 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 36daf303b10940278e2db95c7b2a1398: Generated new TSK 0
I20260620 11:46:14.293454 24099 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.6.234.193@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.6.234.193@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.6.234.193 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task1JRg7N/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781955746769051-7083-0/krb5kdc/kudu_127.6.234.193.keytab.
Entry for principal kudu/127.6.234.193 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task1JRg7N/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781955746769051-7083-0/krb5kdc/kudu_127.6.234.193.keytab.
I20260620 11:46:14.317744 7083 external_mini_cluster.cc:1326] Running /tmp/dist-test-task1JRg7N/build/release/bin/kudu
/tmp/dist-test-task1JRg7N/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task1JRg7N/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781955746769051-7083-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-task1JRg7N/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781955746769051-7083-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task1JRg7N/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781955746769051-7083-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-task1JRg7N/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781955746769051-7083-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.6.234.193:0
--local_ip_for_outbound_sockets=127.6.234.193
--webserver_interface=127.6.234.193
--webserver_port=0
--tserver_master_addrs=127.6.234.254:38591
--keytab_file=/tmp/dist-test-task1JRg7N/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781955746769051-7083-0/krb5kdc/kudu_127.6.234.193.keytab
--principal=kudu/127.6.234.193
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.6.234.212:37185
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task1JRg7N/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781955746769051-7083-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task1JRg7N/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781955746769051-7083-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task1JRg7N/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781955746769051-7083-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 11:46:14.398624 24114 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 11:46:14.398828 24114 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 11:46:14.398850 24114 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 11:46:14.400008 24114 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 11:46:14.400051 24114 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 11:46:14.400077 24114 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.6.234.193
I20260620 11:46:14.401590 24114 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.6.234.212:37185
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task1JRg7N/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781955746769051-7083-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-task1JRg7N/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781955746769051-7083-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task1JRg7N/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781955746769051-7083-0/krb5kdc/kudu_127.6.234.193.keytab
--principal=kudu/127.6.234.193
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.6.234.193:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task1JRg7N/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781955746769051-7083-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.6.234.193
--webserver_port=0
--tserver_master_addrs=127.6.234.254:38591
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.24114
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.6.234.193
--log_dir=/tmp/dist-test-task1JRg7N/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781955746769051-7083-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 20 Jun 2026 10:32:37 UTC on ccycloud.quasar-bgyofv.root.comops.site
build id 4457173
I20260620 11:46:14.401849 24114 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 11:46:14.402093 24114 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 11:46:14.406934 24119 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 11:46:14.406958 24120 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 11:46:14.406934 24122 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 11:46:14.407220 24114 server_base.cc:1034] running on GCE node
I20260620 11:46:14.407466 24114 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 11:46:14.407713 24114 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 11:46:14.408876 24114 hybrid_clock.cc:648] HybridClock initialized: now 1781955974408861 us; error 26 us; skew 500 ppm
Jun 20 11:46:14 dist-test-slave-cjc6 krb5kdc[24007](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781955974, etypes {rep=17 tkt=17 ses=17}, kudu/127.6.234.193@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 11:46:14.411464 24114 init.cc:377] Logged in from keytab as kudu/127.6.234.193@KRBTEST.COM (short username kudu)
I20260620 11:46:14.411957 24114 webserver.cc:460] Webserver started at http://127.6.234.193:41999/ using document root <none> and password file <none>
I20260620 11:46:14.412189 24114 fs_manager.cc:359] Metadata directory not provided
I20260620 11:46:14.412256 24114 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 11:46:14.412408 24114 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 11:46:14.413475 24114 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task1JRg7N/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781955746769051-7083-0/minicluster-data/ts-0/data/instance:
uuid: "5c7e6a52fab64d1788f4717a0dd47c6c"
format_stamp: "Formatted at 2026-06-20 11:46:14 on dist-test-slave-cjc6"
server_key: "c6702ce220038737a9dc0f767dc8333a"
server_key_iv: "36c6698993c223303e80a6db41c081e9"
server_key_version: "encryptionkey@0"
I20260620 11:46:14.413878 24114 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task1JRg7N/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781955746769051-7083-0/minicluster-data/ts-0/wal/instance:
uuid: "5c7e6a52fab64d1788f4717a0dd47c6c"
format_stamp: "Formatted at 2026-06-20 11:46:14 on dist-test-slave-cjc6"
server_key: "c6702ce220038737a9dc0f767dc8333a"
server_key_iv: "36c6698993c223303e80a6db41c081e9"
server_key_version: "encryptionkey@0"
I20260620 11:46:14.415205 24114 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.001s
I20260620 11:46:14.415936 24129 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 11:46:14.416136 24114 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.001s
I20260620 11:46:14.416234 24114 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task1JRg7N/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781955746769051-7083-0/minicluster-data/ts-0/data,/tmp/dist-test-task1JRg7N/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781955746769051-7083-0/minicluster-data/ts-0/wal
uuid: "5c7e6a52fab64d1788f4717a0dd47c6c"
format_stamp: "Formatted at 2026-06-20 11:46:14 on dist-test-slave-cjc6"
server_key: "c6702ce220038737a9dc0f767dc8333a"
server_key_iv: "36c6698993c223303e80a6db41c081e9"
server_key_version: "encryptionkey@0"
I20260620 11:46:14.416319 24114 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task1JRg7N/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781955746769051-7083-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-task1JRg7N/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781955746769051-7083-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-task1JRg7N/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781955746769051-7083-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 11:46:14.437224 24114 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 11:46:14.439615 24114 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 11:46:14.439796 24114 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 11:46:14.440053 24114 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 11:46:14.440376 24114 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 11:46:14.440439 24114 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 11:46:14.440485 24114 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 11:46:14.440510 24114 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 11:46:14.446882 24114 rpc_server.cc:307] RPC server started. Bound to: 127.6.234.193:33071
I20260620 11:46:14.446950 24242 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.6.234.193:33071 every 8 connection(s)
I20260620 11:46:14.447296 24114 server_base.cc:1166] Dumped server information to /tmp/dist-test-task1JRg7N/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781955746769051-7083-0/minicluster-data/ts-0/data/info.pb
I20260620 11:46:14.452364 7083 external_mini_cluster.cc:1388] Started /tmp/dist-test-task1JRg7N/build/release/bin/kudu as pid 24114
I20260620 11:46:14.452490 7083 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task1JRg7N/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781955746769051-7083-0/minicluster-data/ts-0/wal/instance
I20260620 11:46:14.452731 7083 external_mini_cluster.cc:1428] Setting key ec5a06c80a29ad1d83f6255c57e21910
Jun 20 11:46:14 dist-test-slave-cjc6 krb5kdc[24007](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781955974, etypes {rep=17 tkt=17 ses=17}, kudu/127.6.234.193@KRBTEST.COM for kudu/127.6.234.254@KRBTEST.COM
I20260620 11:46:14.457404 24092 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 11:46:14.448049 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.6.234.193:45563 (local address 127.6.234.254:38591)
0620 11:46:14.448101 (+ 52us) server_negotiation.cc:206] Beginning negotiation
0620 11:46:14.448104 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0620 11:46:14.448289 (+ 185us) server_negotiation.cc:407] Connection header received
0620 11:46:14.448829 (+ 540us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 11:46:14.448831 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 11:46:14.448844 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 11:46:14.448902 (+ 58us) server_negotiation.cc:226] Negotiated authn=SASL
0620 11:46:14.449893 (+ 991us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 11:46:14.450411 (+ 518us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 11:46:14.451464 (+ 1053us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 11:46:14.451679 (+ 215us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 11:46:14.451753 (+ 74us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 11:46:14.454261 (+ 2508us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 11:46:14.454263 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 11:46:14.454263 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 11:46:14.454271 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0620 11:46:14.455747 (+ 1476us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 11:46:14.456270 (+ 523us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 11:46:14.456271 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 11:46:14.456280 (+ 9us) server_negotiation.cc:967] calling sasl_server_step()
0620 11:46:14.456346 (+ 66us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 11:46:14.456677 (+ 331us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 11:46:14.456678 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 11:46:14.456679 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 11:46:14.456800 (+ 121us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 11:46:14.456866 (+ 66us) server_negotiation.cc:1035] Waiting for connection context
0620 11:46:14.457227 (+ 361us) server_negotiation.cc:299] Negotiation successful
0620 11:46:14.457303 (+ 76us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":24}
I20260620 11:46:14.457791 24245 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 11:46:14.448010 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.234.254:38591 (local address 127.6.234.193:45563)
0620 11:46:14.448149 (+ 139us) negotiation.cc:107] Waiting for socket to connect
0620 11:46:14.448198 (+ 49us) client_negotiation.cc:174] Beginning negotiation
0620 11:46:14.448717 (+ 519us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 11:46:14.448979 (+ 262us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 11:46:14.448983 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 11:46:14.449564 (+ 581us) client_negotiation.cc:189] Negotiated authn=SASL
0620 11:46:14.449783 (+ 219us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 11:46:14.449787 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 11:46:14.450511 (+ 724us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 11:46:14.450512 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 11:46:14.451342 (+ 830us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 11:46:14.451344 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 11:46:14.451774 (+ 430us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 11:46:14.451775 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 11:46:14.451909 (+ 134us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 11:46:14.452442 (+ 533us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 11:46:14.452454 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0620 11:46:14.454172 (+ 1718us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 11:46:14.455847 (+ 1675us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 11:46:14.455854 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 11:46:14.455856 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0620 11:46:14.456182 (+ 326us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 11:46:14.456424 (+ 242us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 11:46:14.456425 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 11:46:14.456425 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0620 11:46:14.456588 (+ 163us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 11:46:14.456876 (+ 288us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 11:46:14.456894 (+ 18us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 11:46:14.457079 (+ 185us) client_negotiation.cc:769] Sending connection context
0620 11:46:14.457205 (+ 126us) client_negotiation.cc:240] Negotiation successful
0620 11:46:14.457401 (+ 196us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":100,"thread_start_us":65,"threads_started":1}
I20260620 11:46:14.458464 24243 heartbeater.cc:347] Connected to a master server at 127.6.234.254:38591
I20260620 11:46:14.458580 24243 heartbeater.cc:464] Registering TS with master...
I20260620 11:46:14.458799 24243 heartbeater.cc:511] Master 127.6.234.254:38591 requested a full tablet report, sending...
I20260620 11:46:14.459358 24052 ts_manager.cc:194] Registered new tserver with Master: 5c7e6a52fab64d1788f4717a0dd47c6c (127.6.234.193:33071)
I20260620 11:46:14.459995 24052 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.6.234.193@KRBTEST.COM'} at 127.6.234.193:45563
WARNING: no policy specified for kudu/127.6.234.194@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.6.234.194@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.6.234.194 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task1JRg7N/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781955746769051-7083-0/krb5kdc/kudu_127.6.234.194.keytab.
Entry for principal kudu/127.6.234.194 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task1JRg7N/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781955746769051-7083-0/krb5kdc/kudu_127.6.234.194.keytab.
I20260620 11:46:14.480232 7083 external_mini_cluster.cc:1326] Running /tmp/dist-test-task1JRg7N/build/release/bin/kudu
/tmp/dist-test-task1JRg7N/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task1JRg7N/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781955746769051-7083-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-task1JRg7N/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781955746769051-7083-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task1JRg7N/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781955746769051-7083-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-task1JRg7N/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781955746769051-7083-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.6.234.194:0
--local_ip_for_outbound_sockets=127.6.234.194
--webserver_interface=127.6.234.194
--webserver_port=0
--tserver_master_addrs=127.6.234.254:38591
--keytab_file=/tmp/dist-test-task1JRg7N/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781955746769051-7083-0/krb5kdc/kudu_127.6.234.194.keytab
--principal=kudu/127.6.234.194
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.6.234.212:37185
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task1JRg7N/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781955746769051-7083-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task1JRg7N/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781955746769051-7083-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task1JRg7N/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781955746769051-7083-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 11:46:14.562855 24248 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 11:46:14.563053 24248 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 11:46:14.563078 24248 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 11:46:14.564195 24248 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 11:46:14.564237 24248 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 11:46:14.564265 24248 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.6.234.194
I20260620 11:46:14.565789 24248 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.6.234.212:37185
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task1JRg7N/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781955746769051-7083-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-task1JRg7N/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781955746769051-7083-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task1JRg7N/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781955746769051-7083-0/krb5kdc/kudu_127.6.234.194.keytab
--principal=kudu/127.6.234.194
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.6.234.194:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task1JRg7N/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781955746769051-7083-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.6.234.194
--webserver_port=0
--tserver_master_addrs=127.6.234.254:38591
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.24248
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.6.234.194
--log_dir=/tmp/dist-test-task1JRg7N/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781955746769051-7083-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 20 Jun 2026 10:32:37 UTC on ccycloud.quasar-bgyofv.root.comops.site
build id 4457173
I20260620 11:46:14.566037 24248 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 11:46:14.566270 24248 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 11:46:14.571302 24254 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 11:46:14.571302 24256 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 11:46:14.571350 24253 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 11:46:14.571668 24248 server_base.cc:1034] running on GCE node
I20260620 11:46:14.571900 24248 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 11:46:14.572101 24248 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 11:46:14.573273 24248 hybrid_clock.cc:648] HybridClock initialized: now 1781955974573250 us; error 37 us; skew 500 ppm
Jun 20 11:46:14 dist-test-slave-cjc6 krb5kdc[24007](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781955974, etypes {rep=17 tkt=17 ses=17}, kudu/127.6.234.194@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 11:46:14.575878 24248 init.cc:377] Logged in from keytab as kudu/127.6.234.194@KRBTEST.COM (short username kudu)
I20260620 11:46:14.576378 24248 webserver.cc:460] Webserver started at http://127.6.234.194:45759/ using document root <none> and password file <none>
I20260620 11:46:14.576617 24248 fs_manager.cc:359] Metadata directory not provided
I20260620 11:46:14.576661 24248 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 11:46:14.576824 24248 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 11:46:14.577950 24248 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task1JRg7N/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781955746769051-7083-0/minicluster-data/ts-1/data/instance:
uuid: "337261b3e250467f8e7b48b0587816e4"
format_stamp: "Formatted at 2026-06-20 11:46:14 on dist-test-slave-cjc6"
server_key: "42bde1f553f136186f618329631dcdb9"
server_key_iv: "8e4fbc2c620db4260bd223a0a4cf5f0e"
server_key_version: "encryptionkey@0"
I20260620 11:46:14.578299 24248 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task1JRg7N/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781955746769051-7083-0/minicluster-data/ts-1/wal/instance:
uuid: "337261b3e250467f8e7b48b0587816e4"
format_stamp: "Formatted at 2026-06-20 11:46:14 on dist-test-slave-cjc6"
server_key: "42bde1f553f136186f618329631dcdb9"
server_key_iv: "8e4fbc2c620db4260bd223a0a4cf5f0e"
server_key_version: "encryptionkey@0"
I20260620 11:46:14.579644 24248 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.001s sys 0.000s
I20260620 11:46:14.580476 24263 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 11:46:14.580660 24248 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260620 11:46:14.580768 24248 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task1JRg7N/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781955746769051-7083-0/minicluster-data/ts-1/data,/tmp/dist-test-task1JRg7N/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781955746769051-7083-0/minicluster-data/ts-1/wal
uuid: "337261b3e250467f8e7b48b0587816e4"
format_stamp: "Formatted at 2026-06-20 11:46:14 on dist-test-slave-cjc6"
server_key: "42bde1f553f136186f618329631dcdb9"
server_key_iv: "8e4fbc2c620db4260bd223a0a4cf5f0e"
server_key_version: "encryptionkey@0"
I20260620 11:46:14.580857 24248 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task1JRg7N/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781955746769051-7083-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-task1JRg7N/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781955746769051-7083-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-task1JRg7N/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781955746769051-7083-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 11:46:14.601152 24248 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 11:46:14.603245 24248 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 11:46:14.603487 24248 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 11:46:14.603749 24248 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 11:46:14.604096 24248 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 11:46:14.604157 24248 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 11:46:14.604203 24248 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 11:46:14.604234 24248 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 11:46:14.610729 24248 rpc_server.cc:307] RPC server started. Bound to: 127.6.234.194:36987
I20260620 11:46:14.610769 24376 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.6.234.194:36987 every 8 connection(s)
I20260620 11:46:14.611140 24248 server_base.cc:1166] Dumped server information to /tmp/dist-test-task1JRg7N/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781955746769051-7083-0/minicluster-data/ts-1/data/info.pb
I20260620 11:46:14.615298 7083 external_mini_cluster.cc:1388] Started /tmp/dist-test-task1JRg7N/build/release/bin/kudu as pid 24248
I20260620 11:46:14.615442 7083 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task1JRg7N/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781955746769051-7083-0/minicluster-data/ts-1/wal/instance
I20260620 11:46:14.615648 7083 external_mini_cluster.cc:1428] Setting key 6897cbdf79db1c32454ba9034937e793
Jun 20 11:46:14 dist-test-slave-cjc6 krb5kdc[24007](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781955974, etypes {rep=17 tkt=17 ses=17}, kudu/127.6.234.194@KRBTEST.COM for kudu/127.6.234.254@KRBTEST.COM
I20260620 11:46:14.621546 24092 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 11:46:14.611932 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.6.234.194:42105 (local address 127.6.234.254:38591)
0620 11:46:14.612010 (+ 78us) server_negotiation.cc:206] Beginning negotiation
0620 11:46:14.612012 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 11:46:14.612231 (+ 219us) server_negotiation.cc:407] Connection header received
0620 11:46:14.612723 (+ 492us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 11:46:14.612724 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 11:46:14.612733 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 11:46:14.612792 (+ 59us) server_negotiation.cc:226] Negotiated authn=SASL
0620 11:46:14.613816 (+ 1024us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 11:46:14.614302 (+ 486us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 11:46:14.615355 (+ 1053us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 11:46:14.615666 (+ 311us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 11:46:14.615739 (+ 73us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 11:46:14.618338 (+ 2599us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 11:46:14.618339 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 11:46:14.618340 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 11:46:14.618348 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0620 11:46:14.620047 (+ 1699us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 11:46:14.620507 (+ 460us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 11:46:14.620508 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 11:46:14.620508 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 11:46:14.620573 (+ 65us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 11:46:14.620829 (+ 256us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 11:46:14.620830 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 11:46:14.620830 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 11:46:14.620982 (+ 152us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 11:46:14.621037 (+ 55us) server_negotiation.cc:1035] Waiting for connection context
0620 11:46:14.621417 (+ 380us) server_negotiation.cc:299] Negotiation successful
0620 11:46:14.621483 (+ 66us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":43}
I20260620 11:46:14.621959 24379 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 11:46:14.611891 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.234.254:38591 (local address 127.6.234.194:42105)
0620 11:46:14.612065 (+ 174us) negotiation.cc:107] Waiting for socket to connect
0620 11:46:14.612110 (+ 45us) client_negotiation.cc:174] Beginning negotiation
0620 11:46:14.612636 (+ 526us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 11:46:14.612848 (+ 212us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 11:46:14.612852 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 11:46:14.613468 (+ 616us) client_negotiation.cc:189] Negotiated authn=SASL
0620 11:46:14.613676 (+ 208us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 11:46:14.613681 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 11:46:14.614388 (+ 707us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 11:46:14.614388 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 11:46:14.615262 (+ 874us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 11:46:14.615264 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 11:46:14.615759 (+ 495us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 11:46:14.615760 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 11:46:14.615944 (+ 184us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 11:46:14.616590 (+ 646us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 11:46:14.616599 (+ 9us) client_negotiation.cc:656] calling sasl_client_start()
0620 11:46:14.618241 (+ 1642us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 11:46:14.620134 (+ 1893us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 11:46:14.620141 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 11:46:14.620144 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0620 11:46:14.620422 (+ 278us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 11:46:14.620646 (+ 224us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 11:46:14.620647 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 11:46:14.620647 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0620 11:46:14.620755 (+ 108us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 11:46:14.621054 (+ 299us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 11:46:14.621062 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 11:46:14.621254 (+ 192us) client_negotiation.cc:769] Sending connection context
0620 11:46:14.621391 (+ 137us) client_negotiation.cc:240] Negotiation successful
0620 11:46:14.621608 (+ 217us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":131,"thread_start_us":99,"threads_started":1}
I20260620 11:46:14.622500 24377 heartbeater.cc:347] Connected to a master server at 127.6.234.254:38591
I20260620 11:46:14.622614 24377 heartbeater.cc:464] Registering TS with master...
I20260620 11:46:14.622783 24377 heartbeater.cc:511] Master 127.6.234.254:38591 requested a full tablet report, sending...
I20260620 11:46:14.623179 24052 ts_manager.cc:194] Registered new tserver with Master: 337261b3e250467f8e7b48b0587816e4 (127.6.234.194:36987)
I20260620 11:46:14.623548 24052 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.6.234.194@KRBTEST.COM'} at 127.6.234.194:42105
WARNING: no policy specified for kudu/127.6.234.195@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.6.234.195@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.6.234.195 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task1JRg7N/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781955746769051-7083-0/krb5kdc/kudu_127.6.234.195.keytab.
Entry for principal kudu/127.6.234.195 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task1JRg7N/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781955746769051-7083-0/krb5kdc/kudu_127.6.234.195.keytab.
I20260620 11:46:14.641865 7083 external_mini_cluster.cc:1326] Running /tmp/dist-test-task1JRg7N/build/release/bin/kudu
/tmp/dist-test-task1JRg7N/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task1JRg7N/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781955746769051-7083-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-task1JRg7N/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781955746769051-7083-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task1JRg7N/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781955746769051-7083-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-task1JRg7N/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781955746769051-7083-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.6.234.195:0
--local_ip_for_outbound_sockets=127.6.234.195
--webserver_interface=127.6.234.195
--webserver_port=0
--tserver_master_addrs=127.6.234.254:38591
--keytab_file=/tmp/dist-test-task1JRg7N/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781955746769051-7083-0/krb5kdc/kudu_127.6.234.195.keytab
--principal=kudu/127.6.234.195
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.6.234.212:37185
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task1JRg7N/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781955746769051-7083-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task1JRg7N/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781955746769051-7083-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task1JRg7N/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781955746769051-7083-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 11:46:14.722344 24382 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 11:46:14.722546 24382 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 11:46:14.722570 24382 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 11:46:14.723726 24382 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 11:46:14.723767 24382 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 11:46:14.723820 24382 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.6.234.195
I20260620 11:46:14.725266 24382 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.6.234.212:37185
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task1JRg7N/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781955746769051-7083-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-task1JRg7N/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781955746769051-7083-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task1JRg7N/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781955746769051-7083-0/krb5kdc/kudu_127.6.234.195.keytab
--principal=kudu/127.6.234.195
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.6.234.195:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task1JRg7N/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781955746769051-7083-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.6.234.195
--webserver_port=0
--tserver_master_addrs=127.6.234.254:38591
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.24382
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.6.234.195
--log_dir=/tmp/dist-test-task1JRg7N/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781955746769051-7083-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 20 Jun 2026 10:32:37 UTC on ccycloud.quasar-bgyofv.root.comops.site
build id 4457173
I20260620 11:46:14.725518 24382 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 11:46:14.725813 24382 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 11:46:14.730669 24390 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 11:46:14.730676 24388 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 11:46:14.730682 24387 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 11:46:14.730986 24382 server_base.cc:1034] running on GCE node
I20260620 11:46:14.731204 24382 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 11:46:14.731411 24382 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 11:46:14.732548 24382 hybrid_clock.cc:648] HybridClock initialized: now 1781955974732535 us; error 29 us; skew 500 ppm
Jun 20 11:46:14 dist-test-slave-cjc6 krb5kdc[24007](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781955974, etypes {rep=17 tkt=17 ses=17}, kudu/127.6.234.195@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 11:46:14.735152 24382 init.cc:377] Logged in from keytab as kudu/127.6.234.195@KRBTEST.COM (short username kudu)
I20260620 11:46:14.735626 24382 webserver.cc:460] Webserver started at http://127.6.234.195:44887/ using document root <none> and password file <none>
I20260620 11:46:14.735863 24382 fs_manager.cc:359] Metadata directory not provided
I20260620 11:46:14.735932 24382 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 11:46:14.736088 24382 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 11:46:14.737135 24382 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task1JRg7N/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781955746769051-7083-0/minicluster-data/ts-2/data/instance:
uuid: "41936892e4f244c4b357396d71fb7bcf"
format_stamp: "Formatted at 2026-06-20 11:46:14 on dist-test-slave-cjc6"
server_key: "182bf557046c6a2aaaeaa28e1ac4bfb3"
server_key_iv: "84bbe9bebdbcd501de25ebc712ae506c"
server_key_version: "encryptionkey@0"
I20260620 11:46:14.737488 24382 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task1JRg7N/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781955746769051-7083-0/minicluster-data/ts-2/wal/instance:
uuid: "41936892e4f244c4b357396d71fb7bcf"
format_stamp: "Formatted at 2026-06-20 11:46:14 on dist-test-slave-cjc6"
server_key: "182bf557046c6a2aaaeaa28e1ac4bfb3"
server_key_iv: "84bbe9bebdbcd501de25ebc712ae506c"
server_key_version: "encryptionkey@0"
I20260620 11:46:14.738936 24382 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260620 11:46:14.739691 24397 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 11:46:14.739874 24382 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260620 11:46:14.739976 24382 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task1JRg7N/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781955746769051-7083-0/minicluster-data/ts-2/data,/tmp/dist-test-task1JRg7N/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781955746769051-7083-0/minicluster-data/ts-2/wal
uuid: "41936892e4f244c4b357396d71fb7bcf"
format_stamp: "Formatted at 2026-06-20 11:46:14 on dist-test-slave-cjc6"
server_key: "182bf557046c6a2aaaeaa28e1ac4bfb3"
server_key_iv: "84bbe9bebdbcd501de25ebc712ae506c"
server_key_version: "encryptionkey@0"
I20260620 11:46:14.740060 24382 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task1JRg7N/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781955746769051-7083-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-task1JRg7N/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781955746769051-7083-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-task1JRg7N/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781955746769051-7083-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 11:46:14.754626 24382 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 11:46:14.756937 24382 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 11:46:14.757102 24382 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 11:46:14.757328 24382 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 11:46:14.757618 24382 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 11:46:14.757673 24382 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 11:46:14.757735 24382 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 11:46:14.757764 24382 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 11:46:14.763631 24382 rpc_server.cc:307] RPC server started. Bound to: 127.6.234.195:46293
I20260620 11:46:14.763687 24510 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.6.234.195:46293 every 8 connection(s)
I20260620 11:46:14.764056 24382 server_base.cc:1166] Dumped server information to /tmp/dist-test-task1JRg7N/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781955746769051-7083-0/minicluster-data/ts-2/data/info.pb
I20260620 11:46:14.766776 7083 external_mini_cluster.cc:1388] Started /tmp/dist-test-task1JRg7N/build/release/bin/kudu as pid 24382
I20260620 11:46:14.766911 7083 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task1JRg7N/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781955746769051-7083-0/minicluster-data/ts-2/wal/instance
I20260620 11:46:14.767098 7083 external_mini_cluster.cc:1428] Setting key 3201df7d2e46400080c088a430ee9599
Jun 20 11:46:14 dist-test-slave-cjc6 krb5kdc[24007](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781955974, etypes {rep=17 tkt=17 ses=17}, kudu/127.6.234.195@KRBTEST.COM for kudu/127.6.234.254@KRBTEST.COM
I20260620 11:46:14.773850 24092 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 11:46:14.764786 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.6.234.195:58351 (local address 127.6.234.254:38591)
0620 11:46:14.764878 (+ 92us) server_negotiation.cc:206] Beginning negotiation
0620 11:46:14.764880 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 11:46:14.765074 (+ 194us) server_negotiation.cc:407] Connection header received
0620 11:46:14.765543 (+ 469us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 11:46:14.765544 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 11:46:14.765559 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 11:46:14.765610 (+ 51us) server_negotiation.cc:226] Negotiated authn=SASL
0620 11:46:14.766597 (+ 987us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 11:46:14.767416 (+ 819us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 11:46:14.768604 (+ 1188us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 11:46:14.768785 (+ 181us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 11:46:14.768842 (+ 57us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 11:46:14.771091 (+ 2249us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 11:46:14.771092 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 11:46:14.771093 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 11:46:14.771100 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0620 11:46:14.772525 (+ 1425us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 11:46:14.773047 (+ 522us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 11:46:14.773047 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 11:46:14.773048 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 11:46:14.773100 (+ 52us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 11:46:14.773299 (+ 199us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 11:46:14.773299 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 11:46:14.773300 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 11:46:14.773391 (+ 91us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 11:46:14.773435 (+ 44us) server_negotiation.cc:1035] Waiting for connection context
0620 11:46:14.773672 (+ 237us) server_negotiation.cc:299] Negotiation successful
0620 11:46:14.773787 (+ 115us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":51}
I20260620 11:46:14.774089 24513 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 11:46:14.764765 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.234.254:38591 (local address 127.6.234.195:58351)
0620 11:46:14.764948 (+ 183us) negotiation.cc:107] Waiting for socket to connect
0620 11:46:14.764987 (+ 39us) client_negotiation.cc:174] Beginning negotiation
0620 11:46:14.765434 (+ 447us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 11:46:14.765671 (+ 237us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 11:46:14.765675 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 11:46:14.766281 (+ 606us) client_negotiation.cc:189] Negotiated authn=SASL
0620 11:46:14.766468 (+ 187us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 11:46:14.766472 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 11:46:14.767526 (+ 1054us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 11:46:14.767527 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 11:46:14.768534 (+ 1007us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 11:46:14.768536 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 11:46:14.768852 (+ 316us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 11:46:14.768852 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 11:46:14.768947 (+ 95us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 11:46:14.769506 (+ 559us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 11:46:14.769514 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0620 11:46:14.770962 (+ 1448us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 11:46:14.772602 (+ 1640us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 11:46:14.772607 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 11:46:14.772609 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0620 11:46:14.772935 (+ 326us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 11:46:14.773162 (+ 227us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 11:46:14.773163 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 11:46:14.773163 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0620 11:46:14.773259 (+ 96us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 11:46:14.773442 (+ 183us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 11:46:14.773446 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 11:46:14.773556 (+ 110us) client_negotiation.cc:769] Sending connection context
0620 11:46:14.773640 (+ 84us) client_negotiation.cc:240] Negotiation successful
0620 11:46:14.773820 (+ 180us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":147,"thread_start_us":108,"threads_started":1}
I20260620 11:46:14.774649 24511 heartbeater.cc:347] Connected to a master server at 127.6.234.254:38591
I20260620 11:46:14.774760 24511 heartbeater.cc:464] Registering TS with master...
I20260620 11:46:14.774936 24511 heartbeater.cc:511] Master 127.6.234.254:38591 requested a full tablet report, sending...
I20260620 11:46:14.775295 24052 ts_manager.cc:194] Registered new tserver with Master: 41936892e4f244c4b357396d71fb7bcf (127.6.234.195:46293)
I20260620 11:46:14.775619 24052 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.6.234.195@KRBTEST.COM'} at 127.6.234.195:58351
I20260620 11:46:14.778934 7083 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260620 11:46:14.786053 24092 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 11:46:14.780263 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37894 (local address 127.6.234.254:38591)
0620 11:46:14.780321 (+ 58us) server_negotiation.cc:206] Beginning negotiation
0620 11:46:14.780323 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 11:46:14.780382 (+ 59us) server_negotiation.cc:407] Connection header received
0620 11:46:14.780418 (+ 36us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 11:46:14.780419 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 11:46:14.780427 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 11:46:14.780482 (+ 55us) server_negotiation.cc:226] Negotiated authn=SASL
0620 11:46:14.780997 (+ 515us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 11:46:14.781461 (+ 464us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 11:46:14.782239 (+ 778us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 11:46:14.782420 (+ 181us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 11:46:14.782478 (+ 58us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 11:46:14.783577 (+ 1099us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 11:46:14.783578 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 11:46:14.783578 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 11:46:14.783584 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0620 11:46:14.785037 (+ 1453us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 11:46:14.785373 (+ 336us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 11:46:14.785374 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 11:46:14.785374 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 11:46:14.785423 (+ 49us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 11:46:14.785558 (+ 135us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 11:46:14.785559 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 11:46:14.785559 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 11:46:14.785685 (+ 126us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 11:46:14.785825 (+ 140us) server_negotiation.cc:1035] Waiting for connection context
0620 11:46:14.785907 (+ 82us) server_negotiation.cc:299] Negotiation successful
0620 11:46:14.785990 (+ 83us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":36}
I20260620 11:46:14.787609 24052 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:37894:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260620 11:46:14.789160 24052 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260620 11:46:14.795754 24520 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 11:46:14.791289 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.234.195:46293 (local address 127.0.0.1:41528)
0620 11:46:14.791935 (+ 646us) negotiation.cc:107] Waiting for socket to connect
0620 11:46:14.791946 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0620 11:46:14.791995 (+ 49us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 11:46:14.792327 (+ 332us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 11:46:14.792327 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 11:46:14.792329 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 11:46:14.792375 (+ 46us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 11:46:14.792376 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 11:46:14.793362 (+ 986us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 11:46:14.793362 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 11:46:14.794587 (+ 1225us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 11:46:14.794589 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 11:46:14.795458 (+ 869us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 11:46:14.795459 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 11:46:14.795543 (+ 84us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 11:46:14.795557 (+ 14us) client_negotiation.cc:769] Sending connection context
0620 11:46:14.795630 (+ 73us) client_negotiation.cc:240] Negotiation successful
0620 11:46:14.795669 (+ 39us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":634,"mutex_wait_us":84,"thread_start_us":51,"threads_started":1}
I20260620 11:46:14.795789 24522 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 11:46:14.791469 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41528 (local address 127.6.234.195:46293)
0620 11:46:14.791738 (+ 269us) server_negotiation.cc:206] Beginning negotiation
0620 11:46:14.791741 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0620 11:46:14.792180 (+ 439us) server_negotiation.cc:407] Connection header received
0620 11:46:14.792224 (+ 44us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 11:46:14.792227 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 11:46:14.792243 (+ 16us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 11:46:14.792329 (+ 86us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 11:46:14.792453 (+ 124us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 11:46:14.793232 (+ 779us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 11:46:14.794690 (+ 1458us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 11:46:14.795358 (+ 668us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 11:46:14.795457 (+ 99us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 11:46:14.795513 (+ 56us) server_negotiation.cc:1035] Waiting for connection context
0620 11:46:14.795661 (+ 148us) server_negotiation.cc:299] Negotiation successful
0620 11:46:14.795708 (+ 47us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":241,"thread_start_us":86,"threads_started":1}
I20260620 11:46:14.796232 24524 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 11:46:14.791542 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49308 (local address 127.6.234.194:36987)
0620 11:46:14.791690 (+ 148us) server_negotiation.cc:206] Beginning negotiation
0620 11:46:14.791696 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0620 11:46:14.792404 (+ 708us) server_negotiation.cc:407] Connection header received
0620 11:46:14.792447 (+ 43us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 11:46:14.792450 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 11:46:14.792464 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 11:46:14.792547 (+ 83us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 11:46:14.792665 (+ 118us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 11:46:14.793580 (+ 915us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 11:46:14.794938 (+ 1358us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 11:46:14.795675 (+ 737us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 11:46:14.795746 (+ 71us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 11:46:14.795799 (+ 53us) server_negotiation.cc:1035] Waiting for connection context
0620 11:46:14.796097 (+ 298us) server_negotiation.cc:299] Negotiation successful
0620 11:46:14.796147 (+ 50us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":112,"thread_start_us":85,"threads_started":1}
I20260620 11:46:14.796533 24523 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 11:46:14.791479 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33598 (local address 127.6.234.193:33071)
0620 11:46:14.791700 (+ 221us) server_negotiation.cc:206] Beginning negotiation
0620 11:46:14.791704 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 11:46:14.791712 (+ 8us) server_negotiation.cc:407] Connection header received
0620 11:46:14.791743 (+ 31us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 11:46:14.791745 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 11:46:14.791757 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 11:46:14.791841 (+ 84us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 11:46:14.792033 (+ 192us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 11:46:14.793197 (+ 1164us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 11:46:14.794864 (+ 1667us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 11:46:14.795360 (+ 496us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 11:46:14.795454 (+ 94us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 11:46:14.795522 (+ 68us) server_negotiation.cc:1035] Waiting for connection context
0620 11:46:14.796357 (+ 835us) server_negotiation.cc:299] Negotiation successful
0620 11:46:14.796448 (+ 91us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":183,"thread_start_us":102,"threads_started":1}
I20260620 11:46:14.796030 24521 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 11:46:14.791311 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.234.194:36987 (local address 127.0.0.1:49308)
0620 11:46:14.792074 (+ 763us) negotiation.cc:107] Waiting for socket to connect
0620 11:46:14.792083 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0620 11:46:14.792123 (+ 40us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 11:46:14.792553 (+ 430us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 11:46:14.792554 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 11:46:14.792556 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 11:46:14.792587 (+ 31us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 11:46:14.792588 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 11:46:14.793683 (+ 1095us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 11:46:14.793685 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 11:46:14.794848 (+ 1163us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 11:46:14.794850 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 11:46:14.795760 (+ 910us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 11:46:14.795761 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 11:46:14.795823 (+ 62us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 11:46:14.795827 (+ 4us) client_negotiation.cc:769] Sending connection context
0620 11:46:14.795887 (+ 60us) client_negotiation.cc:240] Negotiation successful
0620 11:46:14.795958 (+ 71us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":744,"mutex_wait_us":2,"thread_start_us":53,"threads_started":1}
I20260620 11:46:14.796290 24519 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 11:46:14.791289 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.234.193:33071 (local address 127.0.0.1:33598)
0620 11:46:14.791452 (+ 163us) negotiation.cc:107] Waiting for socket to connect
0620 11:46:14.791494 (+ 42us) client_negotiation.cc:174] Beginning negotiation
0620 11:46:14.791561 (+ 67us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 11:46:14.791863 (+ 302us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 11:46:14.791866 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 11:46:14.791868 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 11:46:14.791934 (+ 66us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 11:46:14.791936 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 11:46:14.793309 (+ 1373us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 11:46:14.793310 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 11:46:14.794753 (+ 1443us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 11:46:14.794756 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 11:46:14.796061 (+ 1305us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 11:46:14.796061 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 11:46:14.796129 (+ 68us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 11:46:14.796132 (+ 3us) client_negotiation.cc:769] Sending connection context
0620 11:46:14.796184 (+ 52us) client_negotiation.cc:240] Negotiation successful
0620 11:46:14.796218 (+ 34us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":139,"mutex_wait_us":111,"thread_start_us":62,"threads_started":1}
I20260620 11:46:14.797319 24445 tablet_service.cc:1467] Processing CreateTablet for tablet 888e3b2abeb342b28f056b7cd0109b1a (DEFAULT_TABLE table=test-table [id=cf5f3e5e757843f793af9073f3958941]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 11:46:14.797508 24311 tablet_service.cc:1467] Processing CreateTablet for tablet 888e3b2abeb342b28f056b7cd0109b1a (DEFAULT_TABLE table=test-table [id=cf5f3e5e757843f793af9073f3958941]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 11:46:14.797567 24177 tablet_service.cc:1467] Processing CreateTablet for tablet 888e3b2abeb342b28f056b7cd0109b1a (DEFAULT_TABLE table=test-table [id=cf5f3e5e757843f793af9073f3958941]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 11:46:14.797736 24445 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 888e3b2abeb342b28f056b7cd0109b1a. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 11:46:14.797861 24311 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 888e3b2abeb342b28f056b7cd0109b1a. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 11:46:14.797912 24177 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 888e3b2abeb342b28f056b7cd0109b1a. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 11:46:14.800231 24525 tablet_bootstrap.cc:492] T 888e3b2abeb342b28f056b7cd0109b1a P 337261b3e250467f8e7b48b0587816e4: Bootstrap starting.
I20260620 11:46:14.800917 24525 tablet_bootstrap.cc:654] T 888e3b2abeb342b28f056b7cd0109b1a P 337261b3e250467f8e7b48b0587816e4: Neither blocks nor log segments found. Creating new log.
I20260620 11:46:14.801003 24526 tablet_bootstrap.cc:492] T 888e3b2abeb342b28f056b7cd0109b1a P 5c7e6a52fab64d1788f4717a0dd47c6c: Bootstrap starting.
I20260620 11:46:14.801592 24525 log.cc:824] T 888e3b2abeb342b28f056b7cd0109b1a P 337261b3e250467f8e7b48b0587816e4: Log is configured to *not* fsync() on all Append() calls
I20260620 11:46:14.801682 24526 tablet_bootstrap.cc:654] T 888e3b2abeb342b28f056b7cd0109b1a P 5c7e6a52fab64d1788f4717a0dd47c6c: Neither blocks nor log segments found. Creating new log.
I20260620 11:46:14.802238 24526 log.cc:824] T 888e3b2abeb342b28f056b7cd0109b1a P 5c7e6a52fab64d1788f4717a0dd47c6c: Log is configured to *not* fsync() on all Append() calls
I20260620 11:46:14.802419 24525 tablet_bootstrap.cc:492] T 888e3b2abeb342b28f056b7cd0109b1a P 337261b3e250467f8e7b48b0587816e4: No bootstrap required, opened a new log
I20260620 11:46:14.802495 24525 ts_tablet_manager.cc:1397] T 888e3b2abeb342b28f056b7cd0109b1a P 337261b3e250467f8e7b48b0587816e4: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20260620 11:46:14.802593 24527 tablet_bootstrap.cc:492] T 888e3b2abeb342b28f056b7cd0109b1a P 41936892e4f244c4b357396d71fb7bcf: Bootstrap starting.
I20260620 11:46:14.802942 24526 tablet_bootstrap.cc:492] T 888e3b2abeb342b28f056b7cd0109b1a P 5c7e6a52fab64d1788f4717a0dd47c6c: No bootstrap required, opened a new log
I20260620 11:46:14.803023 24526 ts_tablet_manager.cc:1397] T 888e3b2abeb342b28f056b7cd0109b1a P 5c7e6a52fab64d1788f4717a0dd47c6c: Time spent bootstrapping tablet: real 0.002s user 0.000s sys 0.002s
I20260620 11:46:14.803217 24527 tablet_bootstrap.cc:654] T 888e3b2abeb342b28f056b7cd0109b1a P 41936892e4f244c4b357396d71fb7bcf: Neither blocks nor log segments found. Creating new log.
I20260620 11:46:14.803799 24527 log.cc:824] T 888e3b2abeb342b28f056b7cd0109b1a P 41936892e4f244c4b357396d71fb7bcf: Log is configured to *not* fsync() on all Append() calls
I20260620 11:46:14.804571 24527 tablet_bootstrap.cc:492] T 888e3b2abeb342b28f056b7cd0109b1a P 41936892e4f244c4b357396d71fb7bcf: No bootstrap required, opened a new log
I20260620 11:46:14.804690 24527 ts_tablet_manager.cc:1397] T 888e3b2abeb342b28f056b7cd0109b1a P 41936892e4f244c4b357396d71fb7bcf: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20260620 11:46:14.804620 24525 raft_consensus.cc:348] T 888e3b2abeb342b28f056b7cd0109b1a P 337261b3e250467f8e7b48b0587816e4 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "41936892e4f244c4b357396d71fb7bcf" member_type: VOTER last_known_addr { host: "127.6.234.195" port: 46293 } } peers { permanent_uuid: "5c7e6a52fab64d1788f4717a0dd47c6c" member_type: VOTER last_known_addr { host: "127.6.234.193" port: 33071 } } peers { permanent_uuid: "337261b3e250467f8e7b48b0587816e4" member_type: VOTER last_known_addr { host: "127.6.234.194" port: 36987 } }
I20260620 11:46:14.804849 24525 raft_consensus.cc:374] T 888e3b2abeb342b28f056b7cd0109b1a P 337261b3e250467f8e7b48b0587816e4 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 11:46:14.804917 24525 raft_consensus.cc:729] T 888e3b2abeb342b28f056b7cd0109b1a P 337261b3e250467f8e7b48b0587816e4 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 337261b3e250467f8e7b48b0587816e4, State: Initialized, Role: FOLLOWER
I20260620 11:46:14.805061 24525 consensus_queue.cc:260] T 888e3b2abeb342b28f056b7cd0109b1a P 337261b3e250467f8e7b48b0587816e4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "41936892e4f244c4b357396d71fb7bcf" member_type: VOTER last_known_addr { host: "127.6.234.195" port: 46293 } } peers { permanent_uuid: "5c7e6a52fab64d1788f4717a0dd47c6c" member_type: VOTER last_known_addr { host: "127.6.234.193" port: 33071 } } peers { permanent_uuid: "337261b3e250467f8e7b48b0587816e4" member_type: VOTER last_known_addr { host: "127.6.234.194" port: 36987 } }
I20260620 11:46:14.804950 24526 raft_consensus.cc:348] T 888e3b2abeb342b28f056b7cd0109b1a P 5c7e6a52fab64d1788f4717a0dd47c6c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "41936892e4f244c4b357396d71fb7bcf" member_type: VOTER last_known_addr { host: "127.6.234.195" port: 46293 } } peers { permanent_uuid: "5c7e6a52fab64d1788f4717a0dd47c6c" member_type: VOTER last_known_addr { host: "127.6.234.193" port: 33071 } } peers { permanent_uuid: "337261b3e250467f8e7b48b0587816e4" member_type: VOTER last_known_addr { host: "127.6.234.194" port: 36987 } }
I20260620 11:46:14.805179 24526 raft_consensus.cc:374] T 888e3b2abeb342b28f056b7cd0109b1a P 5c7e6a52fab64d1788f4717a0dd47c6c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 11:46:14.805244 24526 raft_consensus.cc:729] T 888e3b2abeb342b28f056b7cd0109b1a P 5c7e6a52fab64d1788f4717a0dd47c6c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5c7e6a52fab64d1788f4717a0dd47c6c, State: Initialized, Role: FOLLOWER
I20260620 11:46:14.805341 24525 ts_tablet_manager.cc:1428] T 888e3b2abeb342b28f056b7cd0109b1a P 337261b3e250467f8e7b48b0587816e4: Time spent starting tablet: real 0.003s user 0.003s sys 0.000s
I20260620 11:46:14.805361 24526 consensus_queue.cc:260] T 888e3b2abeb342b28f056b7cd0109b1a P 5c7e6a52fab64d1788f4717a0dd47c6c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "41936892e4f244c4b357396d71fb7bcf" member_type: VOTER last_known_addr { host: "127.6.234.195" port: 46293 } } peers { permanent_uuid: "5c7e6a52fab64d1788f4717a0dd47c6c" member_type: VOTER last_known_addr { host: "127.6.234.193" port: 33071 } } peers { permanent_uuid: "337261b3e250467f8e7b48b0587816e4" member_type: VOTER last_known_addr { host: "127.6.234.194" port: 36987 } }
I20260620 11:46:14.805472 24377 heartbeater.cc:503] Master 127.6.234.254:38591 was elected leader, sending a full tablet report...
I20260620 11:46:14.805661 24526 ts_tablet_manager.cc:1428] T 888e3b2abeb342b28f056b7cd0109b1a P 5c7e6a52fab64d1788f4717a0dd47c6c: Time spent starting tablet: real 0.003s user 0.000s sys 0.003s
I20260620 11:46:14.805680 24243 heartbeater.cc:503] Master 127.6.234.254:38591 was elected leader, sending a full tablet report...
I20260620 11:46:14.807087 24527 raft_consensus.cc:348] T 888e3b2abeb342b28f056b7cd0109b1a P 41936892e4f244c4b357396d71fb7bcf [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "41936892e4f244c4b357396d71fb7bcf" member_type: VOTER last_known_addr { host: "127.6.234.195" port: 46293 } } peers { permanent_uuid: "5c7e6a52fab64d1788f4717a0dd47c6c" member_type: VOTER last_known_addr { host: "127.6.234.193" port: 33071 } } peers { permanent_uuid: "337261b3e250467f8e7b48b0587816e4" member_type: VOTER last_known_addr { host: "127.6.234.194" port: 36987 } }
I20260620 11:46:14.807214 24527 raft_consensus.cc:374] T 888e3b2abeb342b28f056b7cd0109b1a P 41936892e4f244c4b357396d71fb7bcf [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 11:46:14.807235 24527 raft_consensus.cc:729] T 888e3b2abeb342b28f056b7cd0109b1a P 41936892e4f244c4b357396d71fb7bcf [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 41936892e4f244c4b357396d71fb7bcf, State: Initialized, Role: FOLLOWER
I20260620 11:46:14.807359 24527 consensus_queue.cc:260] T 888e3b2abeb342b28f056b7cd0109b1a P 41936892e4f244c4b357396d71fb7bcf [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "41936892e4f244c4b357396d71fb7bcf" member_type: VOTER last_known_addr { host: "127.6.234.195" port: 46293 } } peers { permanent_uuid: "5c7e6a52fab64d1788f4717a0dd47c6c" member_type: VOTER last_known_addr { host: "127.6.234.193" port: 33071 } } peers { permanent_uuid: "337261b3e250467f8e7b48b0587816e4" member_type: VOTER last_known_addr { host: "127.6.234.194" port: 36987 } }
I20260620 11:46:14.807618 24527 ts_tablet_manager.cc:1428] T 888e3b2abeb342b28f056b7cd0109b1a P 41936892e4f244c4b357396d71fb7bcf: Time spent starting tablet: real 0.003s user 0.002s sys 0.000s
I20260620 11:46:14.807672 24511 heartbeater.cc:503] Master 127.6.234.254:38591 was elected leader, sending a full tablet report...
I20260620 11:46:14.822810 24532 raft_consensus.cc:482] T 888e3b2abeb342b28f056b7cd0109b1a P 5c7e6a52fab64d1788f4717a0dd47c6c [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260620 11:46:14.822911 24532 raft_consensus.cc:504] T 888e3b2abeb342b28f056b7cd0109b1a P 5c7e6a52fab64d1788f4717a0dd47c6c [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "41936892e4f244c4b357396d71fb7bcf" member_type: VOTER last_known_addr { host: "127.6.234.195" port: 46293 } } peers { permanent_uuid: "5c7e6a52fab64d1788f4717a0dd47c6c" member_type: VOTER last_known_addr { host: "127.6.234.193" port: 33071 } } peers { permanent_uuid: "337261b3e250467f8e7b48b0587816e4" member_type: VOTER last_known_addr { host: "127.6.234.194" port: 36987 } }
I20260620 11:46:14.823203 24532 leader_election.cc:288] T 888e3b2abeb342b28f056b7cd0109b1a P 5c7e6a52fab64d1788f4717a0dd47c6c [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 41936892e4f244c4b357396d71fb7bcf (127.6.234.195:46293), 337261b3e250467f8e7b48b0587816e4 (127.6.234.194:36987)
I20260620 11:46:14.827059 24522 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 11:46:14.823444 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.6.234.193:44701 (local address 127.6.234.195:46293)
0620 11:46:14.823515 (+ 71us) server_negotiation.cc:206] Beginning negotiation
0620 11:46:14.823517 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 11:46:14.823607 (+ 90us) server_negotiation.cc:407] Connection header received
0620 11:46:14.823691 (+ 84us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 11:46:14.823692 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 11:46:14.823700 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 11:46:14.823744 (+ 44us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 11:46:14.823879 (+ 135us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 11:46:14.824575 (+ 696us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 11:46:14.826123 (+ 1548us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 11:46:14.826700 (+ 577us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 11:46:14.826778 (+ 78us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 11:46:14.826787 (+ 9us) server_negotiation.cc:1035] Waiting for connection context
0620 11:46:14.826952 (+ 165us) server_negotiation.cc:299] Negotiation successful
0620 11:46:14.826991 (+ 39us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":50}
I20260620 11:46:14.827059 24524 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 11:46:14.823465 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.6.234.193:45823 (local address 127.6.234.194:36987)
0620 11:46:14.823514 (+ 49us) server_negotiation.cc:206] Beginning negotiation
0620 11:46:14.823517 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0620 11:46:14.823529 (+ 12us) server_negotiation.cc:407] Connection header received
0620 11:46:14.823563 (+ 34us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 11:46:14.823563 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 11:46:14.823570 (+ 7us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 11:46:14.823653 (+ 83us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 11:46:14.823822 (+ 169us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 11:46:14.824579 (+ 757us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 11:46:14.826123 (+ 1544us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 11:46:14.826699 (+ 576us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 11:46:14.826779 (+ 80us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 11:46:14.826787 (+ 8us) server_negotiation.cc:1035] Waiting for connection context
0620 11:46:14.826952 (+ 165us) server_negotiation.cc:299] Negotiation successful
0620 11:46:14.826991 (+ 39us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":16}
I20260620 11:46:14.827078 24245 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 11:46:14.823370 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.234.194:36987 (local address 127.6.234.193:45823)
0620 11:46:14.823452 (+ 82us) negotiation.cc:107] Waiting for socket to connect
0620 11:46:14.823464 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0620 11:46:14.823507 (+ 43us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 11:46:14.823670 (+ 163us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 11:46:14.823671 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 11:46:14.823673 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 11:46:14.823746 (+ 73us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 11:46:14.823748 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 11:46:14.824670 (+ 922us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 11:46:14.824671 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 11:46:14.826015 (+ 1344us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 11:46:14.826019 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 11:46:14.826788 (+ 769us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 11:46:14.826789 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 11:46:14.826876 (+ 87us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 11:46:14.826878 (+ 2us) client_negotiation.cc:769] Sending connection context
0620 11:46:14.826932 (+ 54us) client_negotiation.cc:240] Negotiation successful
0620 11:46:14.826981 (+ 49us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":62,"mutex_wait_us":78}
I20260620 11:46:14.827086 24534 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 11:46:14.823370 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.234.195:46293 (local address 127.6.234.193:44701)
0620 11:46:14.823539 (+ 169us) negotiation.cc:107] Waiting for socket to connect
0620 11:46:14.823548 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0620 11:46:14.823599 (+ 51us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 11:46:14.823757 (+ 158us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 11:46:14.823757 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 11:46:14.823759 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 11:46:14.823819 (+ 60us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 11:46:14.823820 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 11:46:14.824670 (+ 850us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 11:46:14.824671 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 11:46:14.826015 (+ 1344us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 11:46:14.826019 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 11:46:14.826789 (+ 770us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 11:46:14.826789 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 11:46:14.826868 (+ 79us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 11:46:14.826872 (+ 4us) client_negotiation.cc:769] Sending connection context
0620 11:46:14.826932 (+ 60us) client_negotiation.cc:240] Negotiation successful
0620 11:46:14.826981 (+ 49us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":135,"mutex_wait_us":56,"thread_start_us":93,"threads_started":1}
I20260620 11:46:14.827471 24331 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "888e3b2abeb342b28f056b7cd0109b1a" candidate_uuid: "5c7e6a52fab64d1788f4717a0dd47c6c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "337261b3e250467f8e7b48b0587816e4" is_pre_election: true
I20260620 11:46:14.827531 24465 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "888e3b2abeb342b28f056b7cd0109b1a" candidate_uuid: "5c7e6a52fab64d1788f4717a0dd47c6c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "41936892e4f244c4b357396d71fb7bcf" is_pre_election: true
I20260620 11:46:14.827641 24331 raft_consensus.cc:2454] T 888e3b2abeb342b28f056b7cd0109b1a P 337261b3e250467f8e7b48b0587816e4 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 5c7e6a52fab64d1788f4717a0dd47c6c in term 0.
I20260620 11:46:14.827661 24465 raft_consensus.cc:2454] T 888e3b2abeb342b28f056b7cd0109b1a P 41936892e4f244c4b357396d71fb7bcf [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 5c7e6a52fab64d1788f4717a0dd47c6c in term 0.
I20260620 11:46:14.827864 24132 leader_election.cc:302] T 888e3b2abeb342b28f056b7cd0109b1a P 5c7e6a52fab64d1788f4717a0dd47c6c [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 337261b3e250467f8e7b48b0587816e4, 5c7e6a52fab64d1788f4717a0dd47c6c; no voters:
I20260620 11:46:14.828027 24532 raft_consensus.cc:2781] T 888e3b2abeb342b28f056b7cd0109b1a P 5c7e6a52fab64d1788f4717a0dd47c6c [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260620 11:46:14.828083 24532 raft_consensus.cc:482] T 888e3b2abeb342b28f056b7cd0109b1a P 5c7e6a52fab64d1788f4717a0dd47c6c [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260620 11:46:14.828135 24532 raft_consensus.cc:3037] T 888e3b2abeb342b28f056b7cd0109b1a P 5c7e6a52fab64d1788f4717a0dd47c6c [term 0 FOLLOWER]: Advancing to term 1
I20260620 11:46:14.828961 24532 raft_consensus.cc:504] T 888e3b2abeb342b28f056b7cd0109b1a P 5c7e6a52fab64d1788f4717a0dd47c6c [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "41936892e4f244c4b357396d71fb7bcf" member_type: VOTER last_known_addr { host: "127.6.234.195" port: 46293 } } peers { permanent_uuid: "5c7e6a52fab64d1788f4717a0dd47c6c" member_type: VOTER last_known_addr { host: "127.6.234.193" port: 33071 } } peers { permanent_uuid: "337261b3e250467f8e7b48b0587816e4" member_type: VOTER last_known_addr { host: "127.6.234.194" port: 36987 } }
I20260620 11:46:14.829115 24532 leader_election.cc:288] T 888e3b2abeb342b28f056b7cd0109b1a P 5c7e6a52fab64d1788f4717a0dd47c6c [CANDIDATE]: Term 1 election: Requested vote from peers 41936892e4f244c4b357396d71fb7bcf (127.6.234.195:46293), 337261b3e250467f8e7b48b0587816e4 (127.6.234.194:36987)
I20260620 11:46:14.829213 24465 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "888e3b2abeb342b28f056b7cd0109b1a" candidate_uuid: "5c7e6a52fab64d1788f4717a0dd47c6c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "41936892e4f244c4b357396d71fb7bcf"
I20260620 11:46:14.829253 24331 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "888e3b2abeb342b28f056b7cd0109b1a" candidate_uuid: "5c7e6a52fab64d1788f4717a0dd47c6c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "337261b3e250467f8e7b48b0587816e4"
I20260620 11:46:14.829308 24465 raft_consensus.cc:3037] T 888e3b2abeb342b28f056b7cd0109b1a P 41936892e4f244c4b357396d71fb7bcf [term 0 FOLLOWER]: Advancing to term 1
I20260620 11:46:14.829329 24331 raft_consensus.cc:3037] T 888e3b2abeb342b28f056b7cd0109b1a P 337261b3e250467f8e7b48b0587816e4 [term 0 FOLLOWER]: Advancing to term 1
I20260620 11:46:14.830242 24465 raft_consensus.cc:2454] T 888e3b2abeb342b28f056b7cd0109b1a P 41936892e4f244c4b357396d71fb7bcf [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 5c7e6a52fab64d1788f4717a0dd47c6c in term 1.
I20260620 11:46:14.830242 24331 raft_consensus.cc:2454] T 888e3b2abeb342b28f056b7cd0109b1a P 337261b3e250467f8e7b48b0587816e4 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 5c7e6a52fab64d1788f4717a0dd47c6c in term 1.
I20260620 11:46:14.830431 24133 leader_election.cc:302] T 888e3b2abeb342b28f056b7cd0109b1a P 5c7e6a52fab64d1788f4717a0dd47c6c [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 41936892e4f244c4b357396d71fb7bcf, 5c7e6a52fab64d1788f4717a0dd47c6c; no voters:
I20260620 11:46:14.830521 24532 raft_consensus.cc:2781] T 888e3b2abeb342b28f056b7cd0109b1a P 5c7e6a52fab64d1788f4717a0dd47c6c [term 1 FOLLOWER]: Leader election won for term 1
I20260620 11:46:14.830669 24532 raft_consensus.cc:686] T 888e3b2abeb342b28f056b7cd0109b1a P 5c7e6a52fab64d1788f4717a0dd47c6c [term 1 LEADER]: Becoming Leader. State: Replica: 5c7e6a52fab64d1788f4717a0dd47c6c, State: Running, Role: LEADER
I20260620 11:46:14.830765 24532 consensus_queue.cc:237] T 888e3b2abeb342b28f056b7cd0109b1a P 5c7e6a52fab64d1788f4717a0dd47c6c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "41936892e4f244c4b357396d71fb7bcf" member_type: VOTER last_known_addr { host: "127.6.234.195" port: 46293 } } peers { permanent_uuid: "5c7e6a52fab64d1788f4717a0dd47c6c" member_type: VOTER last_known_addr { host: "127.6.234.193" port: 33071 } } peers { permanent_uuid: "337261b3e250467f8e7b48b0587816e4" member_type: VOTER last_known_addr { host: "127.6.234.194" port: 36987 } }
I20260620 11:46:14.831575 24052 catalog_manager.cc:5495] T 888e3b2abeb342b28f056b7cd0109b1a P 5c7e6a52fab64d1788f4717a0dd47c6c reported cstate change: term changed from 0 to 1, leader changed from <none> to 5c7e6a52fab64d1788f4717a0dd47c6c (127.6.234.193). New cstate: current_term: 1 leader_uuid: "5c7e6a52fab64d1788f4717a0dd47c6c" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "41936892e4f244c4b357396d71fb7bcf" member_type: VOTER last_known_addr { host: "127.6.234.195" port: 46293 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "5c7e6a52fab64d1788f4717a0dd47c6c" member_type: VOTER last_known_addr { host: "127.6.234.193" port: 33071 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "337261b3e250467f8e7b48b0587816e4" member_type: VOTER last_known_addr { host: "127.6.234.194" port: 36987 } health_report { overall_health: UNKNOWN } } }
I20260620 11:46:14.842281 24523 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 11:46:14.839874 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33612 (local address 127.6.234.193:33071)
0620 11:46:14.839927 (+ 53us) server_negotiation.cc:206] Beginning negotiation
0620 11:46:14.839929 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 11:46:14.839938 (+ 9us) server_negotiation.cc:407] Connection header received
0620 11:46:14.839982 (+ 44us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 11:46:14.839983 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 11:46:14.840008 (+ 25us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 11:46:14.840091 (+ 83us) server_negotiation.cc:226] Negotiated authn=TOKEN
0620 11:46:14.840239 (+ 148us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 11:46:14.840704 (+ 465us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 11:46:14.841468 (+ 764us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 11:46:14.841679 (+ 211us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 11:46:14.841782 (+ 103us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 11:46:14.841964 (+ 182us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0620 11:46:14.842069 (+ 105us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0620 11:46:14.842163 (+ 94us) server_negotiation.cc:1035] Waiting for connection context
0620 11:46:14.842187 (+ 24us) server_negotiation.cc:299] Negotiation successful
0620 11:46:14.842206 (+ 19us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":30}
I20260620 11:46:14.844147 24331 raft_consensus.cc:1261] T 888e3b2abeb342b28f056b7cd0109b1a P 337261b3e250467f8e7b48b0587816e4 [term 1 FOLLOWER]: Refusing update from remote peer 5c7e6a52fab64d1788f4717a0dd47c6c: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 11:46:14.844147 24465 raft_consensus.cc:1261] T 888e3b2abeb342b28f056b7cd0109b1a P 41936892e4f244c4b357396d71fb7bcf [term 1 FOLLOWER]: Refusing update from remote peer 5c7e6a52fab64d1788f4717a0dd47c6c: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 11:46:14.844540 24535 consensus_queue.cc:1035] T 888e3b2abeb342b28f056b7cd0109b1a P 5c7e6a52fab64d1788f4717a0dd47c6c [LEADER]: Connected to new peer: Peer: permanent_uuid: "337261b3e250467f8e7b48b0587816e4" member_type: VOTER last_known_addr { host: "127.6.234.194" port: 36987 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 11:46:14.844693 24532 consensus_queue.cc:1035] T 888e3b2abeb342b28f056b7cd0109b1a P 5c7e6a52fab64d1788f4717a0dd47c6c [LEADER]: Connected to new peer: Peer: permanent_uuid: "41936892e4f244c4b357396d71fb7bcf" member_type: VOTER last_known_addr { host: "127.6.234.195" port: 46293 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 11:46:14.846894 24538 mvcc.cc:205] Tried to move back new op lower bound from 7298891672958685184 to 7298891672907493376. Current Snapshot: MvccSnapshot[applied={T|T < 7298891672958685184}]
I20260620 11:46:14.847038 24540 mvcc.cc:205] Tried to move back new op lower bound from 7298891672958685184 to 7298891672907493376. Current Snapshot: MvccSnapshot[applied={T|T < 7298891672958685184}]
I20260620 11:46:14.849728 24052 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:37894:
table { table_name: "test-table" } modify_external_catalogs: true
I20260620 11:46:14.849840 24052 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:37894:
table { table_name: "test-table" } modify_external_catalogs: true
I20260620 11:46:14.850610 24052 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 36daf303b10940278e2db95c7b2a1398: Sending DeleteTablet for 3 replicas of tablet 888e3b2abeb342b28f056b7cd0109b1a
I20260620 11:46:14.850939 24177 tablet_service.cc:1514] Processing DeleteTablet for tablet 888e3b2abeb342b28f056b7cd0109b1a with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 11:46:14 UTC) from {username='kudu', principal='kudu/127.6.234.254@KRBTEST.COM'} at 127.0.0.1:33598
I20260620 11:46:14.850951 24445 tablet_service.cc:1514] Processing DeleteTablet for tablet 888e3b2abeb342b28f056b7cd0109b1a with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 11:46:14 UTC) from {username='kudu', principal='kudu/127.6.234.254@KRBTEST.COM'} at 127.0.0.1:41528
I20260620 11:46:14.851166 24548 tablet_replica.cc:330] T 888e3b2abeb342b28f056b7cd0109b1a P 41936892e4f244c4b357396d71fb7bcf: stopping tablet replica
I20260620 11:46:14.851168 24311 tablet_service.cc:1514] Processing DeleteTablet for tablet 888e3b2abeb342b28f056b7cd0109b1a with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 11:46:14 UTC) from {username='kudu', principal='kudu/127.6.234.254@KRBTEST.COM'} at 127.0.0.1:49308
I20260620 11:46:14.851248 24549 tablet_replica.cc:330] T 888e3b2abeb342b28f056b7cd0109b1a P 5c7e6a52fab64d1788f4717a0dd47c6c: stopping tablet replica
I20260620 11:46:14.851351 24548 raft_consensus.cc:2229] T 888e3b2abeb342b28f056b7cd0109b1a P 41936892e4f244c4b357396d71fb7bcf [term 1 FOLLOWER]: Raft consensus shutting down.
I20260620 11:46:14.851379 24549 raft_consensus.cc:2229] T 888e3b2abeb342b28f056b7cd0109b1a P 5c7e6a52fab64d1788f4717a0dd47c6c [term 1 LEADER]: Raft consensus shutting down.
I20260620 11:46:14.851497 24548 raft_consensus.cc:2258] T 888e3b2abeb342b28f056b7cd0109b1a P 41936892e4f244c4b357396d71fb7bcf [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 11:46:14.851521 24549 raft_consensus.cc:2258] T 888e3b2abeb342b28f056b7cd0109b1a P 5c7e6a52fab64d1788f4717a0dd47c6c [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 11:46:14.851627 7083 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task1JRg7N/build/release/bin/kudu with pid 24114
I20260620 11:46:14.852816 24548 ts_tablet_manager.cc:1905] T 888e3b2abeb342b28f056b7cd0109b1a P 41936892e4f244c4b357396d71fb7bcf: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 11:46:14.854974 24548 ts_tablet_manager.cc:1918] T 888e3b2abeb342b28f056b7cd0109b1a P 41936892e4f244c4b357396d71fb7bcf: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260620 11:46:14.855074 24548 log.cc:1196] T 888e3b2abeb342b28f056b7cd0109b1a P 41936892e4f244c4b357396d71fb7bcf: Deleting WAL directory at /tmp/dist-test-task1JRg7N/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781955746769051-7083-0/minicluster-data/ts-2/wal/wals/888e3b2abeb342b28f056b7cd0109b1a
I20260620 11:46:14.855319 24548 ts_tablet_manager.cc:1939] T 888e3b2abeb342b28f056b7cd0109b1a P 41936892e4f244c4b357396d71fb7bcf: Deleting consensus metadata
I20260620 11:46:14.855731 24039 catalog_manager.cc:4841] TS 41936892e4f244c4b357396d71fb7bcf (127.6.234.195:46293): tablet 888e3b2abeb342b28f056b7cd0109b1a (table test-table [id=cf5f3e5e757843f793af9073f3958941]) successfully deleted
W20260620 11:46:14.856911 24037 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.6.234.193:33071 (error 108)
I20260620 11:46:14.857406 24520 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 11:46:14.857265 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.234.193:33071 (local address 127.0.0.1:33626)
0620 11:46:14.857327 (+ 62us) negotiation.cc:107] Waiting for socket to connect
0620 11:46:14.857373 (+ 46us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.234.193:33071: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":40}
I20260620 11:46:14.857393 24550 tablet_replica.cc:330] T 888e3b2abeb342b28f056b7cd0109b1a P 337261b3e250467f8e7b48b0587816e4: stopping tablet replica
I20260620 11:46:14.857537 24550 raft_consensus.cc:2229] T 888e3b2abeb342b28f056b7cd0109b1a P 337261b3e250467f8e7b48b0587816e4 [term 1 FOLLOWER]: Raft consensus shutting down.
W20260620 11:46:14.857533 24037 catalog_manager.cc:4584] TS 5c7e6a52fab64d1788f4717a0dd47c6c (127.6.234.193:33071): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 888e3b2abeb342b28f056b7cd0109b1a: Network error: Client connection negotiation failed: client connection to 127.6.234.193:33071: connect: Connection refused (error 111)
I20260620 11:46:14.857669 24550 raft_consensus.cc:2258] T 888e3b2abeb342b28f056b7cd0109b1a P 337261b3e250467f8e7b48b0587816e4 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 11:46:14.857882 7083 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task1JRg7N/build/release/bin/kudu with pid 24248
I20260620 11:46:14.862736 7083 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task1JRg7N/build/release/bin/kudu with pid 24382
I20260620 11:46:14.862761 24520 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 11:46:14.862650 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.234.194:36987 (local address 127.0.0.1:49314)
0620 11:46:14.862705 (+ 55us) negotiation.cc:107] Waiting for socket to connect
0620 11:46:14.862736 (+ 31us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.234.194:36987: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":37}
I20260620 11:46:14.867106 7083 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task1JRg7N/build/release/bin/kudu with pid 24020
2026-06-20T11:46:14Z chronyd exiting
[ OK ] EncryptionPolicyTest.TestEncryptionPolicy/0 (1878 ms)
[ RUN ] EncryptionPolicyTest.TestEncryptionPolicy/1
2026-06-20T11:46:14Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-20T11:46:14Z Disabled control of system clock
I20260620 11:46:14.894209 7083 external_mini_cluster.cc:1326] Running /tmp/dist-test-task1JRg7N/build/release/bin/kudu
/tmp/dist-test-task1JRg7N/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task1JRg7N/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781955746769051-7083-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task1JRg7N/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781955746769051-7083-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task1JRg7N/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781955746769051-7083-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task1JRg7N/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781955746769051-7083-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.6.234.254:44741
--webserver_interface=127.6.234.254
--webserver_port=0
--builtin_ntp_servers=127.6.234.212:34529
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.6.234.254:44741
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
W20260620 11:46:14.974267 24555 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 11:46:14.974478 24555 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 11:46:14.974504 24555 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 11:46:14.975689 24555 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 11:46:14.975732 24555 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 11:46:14.975745 24555 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 11:46:14.975796 24555 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 11:46:14.975822 24555 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 11:46:14.977174 24555 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.6.234.212:34529
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task1JRg7N/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781955746769051-7083-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task1JRg7N/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781955746769051-7083-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.6.234.254:44741
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.6.234.254:44741
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task1JRg7N/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781955746769051-7083-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.6.234.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.24555
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task1JRg7N/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781955746769051-7083-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 20 Jun 2026 10:32:37 UTC on ccycloud.quasar-bgyofv.root.comops.site
build id 4457173
I20260620 11:46:14.977437 24555 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 11:46:14.977785 24555 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 11:46:14.982570 24561 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 11:46:14.982581 24563 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 11:46:14.982664 24555 server_base.cc:1034] running on GCE node
W20260620 11:46:14.982604 24560 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 11:46:14.983112 24555 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 11:46:14.983407 24555 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 11:46:14.984583 24555 hybrid_clock.cc:648] HybridClock initialized: now 1781955974984575 us; error 35 us; skew 500 ppm
I20260620 11:46:14.985956 24555 webserver.cc:460] Webserver started at http://127.6.234.254:34219/ using document root <none> and password file <none>
I20260620 11:46:14.986199 24555 fs_manager.cc:359] Metadata directory not provided
I20260620 11:46:14.986270 24555 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 11:46:14.986431 24555 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 11:46:14.987528 24555 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task1JRg7N/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781955746769051-7083-0/minicluster-data/master-0/data/instance:
uuid: "4bee377068c6457ba3268e27a513012c"
format_stamp: "Formatted at 2026-06-20 11:46:14 on dist-test-slave-cjc6"
server_key: "7741f12c1614d589a43bb55dc5855834"
server_key_iv: "21e043d8cdb73d680f3d33bc5b49c9da"
server_key_version: "encryptionkey@0"
I20260620 11:46:14.987912 24555 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task1JRg7N/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781955746769051-7083-0/minicluster-data/master-0/wal/instance:
uuid: "4bee377068c6457ba3268e27a513012c"
format_stamp: "Formatted at 2026-06-20 11:46:14 on dist-test-slave-cjc6"
server_key: "7741f12c1614d589a43bb55dc5855834"
server_key_iv: "21e043d8cdb73d680f3d33bc5b49c9da"
server_key_version: "encryptionkey@0"
I20260620 11:46:14.989184 24555 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260620 11:46:14.990006 24569 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 11:46:14.990191 24555 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260620 11:46:14.990288 24555 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task1JRg7N/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781955746769051-7083-0/minicluster-data/master-0/data,/tmp/dist-test-task1JRg7N/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781955746769051-7083-0/minicluster-data/master-0/wal
uuid: "4bee377068c6457ba3268e27a513012c"
format_stamp: "Formatted at 2026-06-20 11:46:14 on dist-test-slave-cjc6"
server_key: "7741f12c1614d589a43bb55dc5855834"
server_key_iv: "21e043d8cdb73d680f3d33bc5b49c9da"
server_key_version: "encryptionkey@0"
I20260620 11:46:14.990370 24555 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task1JRg7N/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781955746769051-7083-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task1JRg7N/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781955746769051-7083-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task1JRg7N/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781955746769051-7083-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 11:46:15.003443 24555 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 11:46:15.003767 24555 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 11:46:15.003927 24555 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 11:46:15.007742 24555 rpc_server.cc:307] RPC server started. Bound to: 127.6.234.254:44741
I20260620 11:46:15.007777 24621 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.6.234.254:44741 every 8 connection(s)
I20260620 11:46:15.008145 24555 server_base.cc:1166] Dumped server information to /tmp/dist-test-task1JRg7N/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781955746769051-7083-0/minicluster-data/master-0/data/info.pb
I20260620 11:46:15.008406 7083 external_mini_cluster.cc:1388] Started /tmp/dist-test-task1JRg7N/build/release/bin/kudu as pid 24555
I20260620 11:46:15.008519 7083 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task1JRg7N/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781955746769051-7083-0/minicluster-data/master-0/wal/instance
I20260620 11:46:15.008746 7083 external_mini_cluster.cc:1428] Setting key 5d6bdb063c3effa38e119f77efaf721e
I20260620 11:46:15.009405 24622 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 11:46:15.012355 24622 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 4bee377068c6457ba3268e27a513012c: Bootstrap starting.
I20260620 11:46:15.012950 24622 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 4bee377068c6457ba3268e27a513012c: Neither blocks nor log segments found. Creating new log.
I20260620 11:46:15.013244 24622 log.cc:824] T 00000000000000000000000000000000 P 4bee377068c6457ba3268e27a513012c: Log is configured to *not* fsync() on all Append() calls
I20260620 11:46:15.013932 24622 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 4bee377068c6457ba3268e27a513012c: No bootstrap required, opened a new log
I20260620 11:46:15.014549 24625 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 11:46:15.009316 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59652 (local address 127.6.234.254:44741)
0620 11:46:15.009518 (+ 202us) server_negotiation.cc:206] Beginning negotiation
0620 11:46:15.009527 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0620 11:46:15.009559 (+ 32us) server_negotiation.cc:407] Connection header received
0620 11:46:15.010062 (+ 503us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 11:46:15.010069 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 11:46:15.010130 (+ 61us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 11:46:15.010245 (+ 115us) server_negotiation.cc:226] Negotiated authn=SASL
0620 11:46:15.010520 (+ 275us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 11:46:15.011430 (+ 910us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 11:46:15.012538 (+ 1108us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 11:46:15.012959 (+ 421us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 11:46:15.013095 (+ 136us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 11:46:15.013848 (+ 753us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 11:46:15.013852 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 11:46:15.013853 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 11:46:15.014080 (+ 227us) server_negotiation.cc:922] calling sasl_server_start()
0620 11:46:15.014121 (+ 41us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 11:46:15.014133 (+ 12us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 11:46:15.014199 (+ 66us) server_negotiation.cc:1035] Waiting for connection context
0620 11:46:15.014286 (+ 87us) server_negotiation.cc:299] Negotiation successful
0620 11:46:15.014364 (+ 78us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":126,"thread_start_us":98,"threads_started":1}
I20260620 11:46:15.015995 24622 raft_consensus.cc:348] T 00000000000000000000000000000000 P 4bee377068c6457ba3268e27a513012c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4bee377068c6457ba3268e27a513012c" member_type: VOTER last_known_addr { host: "127.6.234.254" port: 44741 } }
I20260620 11:46:15.016130 24622 raft_consensus.cc:374] T 00000000000000000000000000000000 P 4bee377068c6457ba3268e27a513012c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 11:46:15.016153 24622 raft_consensus.cc:729] T 00000000000000000000000000000000 P 4bee377068c6457ba3268e27a513012c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4bee377068c6457ba3268e27a513012c, State: Initialized, Role: FOLLOWER
I20260620 11:46:15.016284 24622 consensus_queue.cc:260] T 00000000000000000000000000000000 P 4bee377068c6457ba3268e27a513012c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4bee377068c6457ba3268e27a513012c" member_type: VOTER last_known_addr { host: "127.6.234.254" port: 44741 } }
I20260620 11:46:15.016352 24622 raft_consensus.cc:388] T 00000000000000000000000000000000 P 4bee377068c6457ba3268e27a513012c [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260620 11:46:15.016407 24622 raft_consensus.cc:482] T 00000000000000000000000000000000 P 4bee377068c6457ba3268e27a513012c [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260620 11:46:15.016454 24622 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 4bee377068c6457ba3268e27a513012c [term 0 FOLLOWER]: Advancing to term 1
I20260620 11:46:15.017027 24622 raft_consensus.cc:504] T 00000000000000000000000000000000 P 4bee377068c6457ba3268e27a513012c [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4bee377068c6457ba3268e27a513012c" member_type: VOTER last_known_addr { host: "127.6.234.254" port: 44741 } }
I20260620 11:46:15.017150 24622 leader_election.cc:302] T 00000000000000000000000000000000 P 4bee377068c6457ba3268e27a513012c [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 4bee377068c6457ba3268e27a513012c; no voters:
I20260620 11:46:15.017333 24622 leader_election.cc:288] T 00000000000000000000000000000000 P 4bee377068c6457ba3268e27a513012c [CANDIDATE]: Term 1 election: Requested vote from peers
I20260620 11:46:15.017369 24627 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 4bee377068c6457ba3268e27a513012c [term 1 FOLLOWER]: Leader election won for term 1
I20260620 11:46:15.017530 24627 raft_consensus.cc:686] T 00000000000000000000000000000000 P 4bee377068c6457ba3268e27a513012c [term 1 LEADER]: Becoming Leader. State: Replica: 4bee377068c6457ba3268e27a513012c, State: Running, Role: LEADER
I20260620 11:46:15.017580 24622 sys_catalog.cc:564] T 00000000000000000000000000000000 P 4bee377068c6457ba3268e27a513012c [sys.catalog]: configured and running, proceeding with master startup.
I20260620 11:46:15.017683 24627 consensus_queue.cc:237] T 00000000000000000000000000000000 P 4bee377068c6457ba3268e27a513012c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4bee377068c6457ba3268e27a513012c" member_type: VOTER last_known_addr { host: "127.6.234.254" port: 44741 } }
I20260620 11:46:15.018067 24628 sys_catalog.cc:455] T 00000000000000000000000000000000 P 4bee377068c6457ba3268e27a513012c [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "4bee377068c6457ba3268e27a513012c" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4bee377068c6457ba3268e27a513012c" member_type: VOTER last_known_addr { host: "127.6.234.254" port: 44741 } } }
I20260620 11:46:15.018075 24629 sys_catalog.cc:455] T 00000000000000000000000000000000 P 4bee377068c6457ba3268e27a513012c [sys.catalog]: SysCatalogTable state changed. Reason: New leader 4bee377068c6457ba3268e27a513012c. Latest consensus state: current_term: 1 leader_uuid: "4bee377068c6457ba3268e27a513012c" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4bee377068c6457ba3268e27a513012c" member_type: VOTER last_known_addr { host: "127.6.234.254" port: 44741 } } }
I20260620 11:46:15.018205 24628 sys_catalog.cc:458] T 00000000000000000000000000000000 P 4bee377068c6457ba3268e27a513012c [sys.catalog]: This master's current role is: LEADER
I20260620 11:46:15.018215 24629 sys_catalog.cc:458] T 00000000000000000000000000000000 P 4bee377068c6457ba3268e27a513012c [sys.catalog]: This master's current role is: LEADER
W20260620 11:46:15.019388 24643 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 4bee377068c6457ba3268e27a513012c: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260620 11:46:15.019450 24643 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260620 11:46:15.019512 24636 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260620 11:46:15.019701 24636 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260620 11:46:15.021270 24636 catalog_manager.cc:1347] Generated new cluster ID: 7e17fb5acc994626a0b6f6a9bc0a0bdf
I20260620 11:46:15.021325 24636 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260620 11:46:15.028661 24636 catalog_manager.cc:1370] Generated new certificate authority record
I20260620 11:46:15.029220 24636 catalog_manager.cc:1504] Loading token signing keys...
I20260620 11:46:15.033579 24636 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 4bee377068c6457ba3268e27a513012c: Generated new TSK 0
I20260620 11:46:15.033772 24636 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260620 11:46:15.036610 7083 external_mini_cluster.cc:1326] Running /tmp/dist-test-task1JRg7N/build/release/bin/kudu
/tmp/dist-test-task1JRg7N/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task1JRg7N/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781955746769051-7083-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-task1JRg7N/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781955746769051-7083-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task1JRg7N/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781955746769051-7083-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-task1JRg7N/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781955746769051-7083-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.6.234.193:0
--local_ip_for_outbound_sockets=127.6.234.193
--webserver_interface=127.6.234.193
--webserver_port=0
--tserver_master_addrs=127.6.234.254:44741
--builtin_ntp_servers=127.6.234.212:34529
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
W20260620 11:46:15.121531 24646 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 11:46:15.121796 24646 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 11:46:15.121835 24646 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 11:46:15.122983 24646 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 11:46:15.123050 24646 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 11:46:15.123093 24646 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.6.234.193
I20260620 11:46:15.124425 24646 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.6.234.212:34529
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task1JRg7N/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781955746769051-7083-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-task1JRg7N/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781955746769051-7083-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.6.234.193:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task1JRg7N/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781955746769051-7083-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.6.234.193
--webserver_port=0
--tserver_master_addrs=127.6.234.254:44741
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.24646
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.6.234.193
--log_dir=/tmp/dist-test-task1JRg7N/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781955746769051-7083-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 20 Jun 2026 10:32:37 UTC on ccycloud.quasar-bgyofv.root.comops.site
build id 4457173
I20260620 11:46:15.124680 24646 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 11:46:15.124977 24646 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 11:46:15.130916 24652 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 11:46:15.130858 24651 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 11:46:15.130858 24654 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 11:46:15.131140 24646 server_base.cc:1034] running on GCE node
I20260620 11:46:15.131409 24646 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 11:46:15.131695 24646 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 11:46:15.132848 24646 hybrid_clock.cc:648] HybridClock initialized: now 1781955975132825 us; error 38 us; skew 500 ppm
I20260620 11:46:15.134239 24646 webserver.cc:460] Webserver started at http://127.6.234.193:39871/ using document root <none> and password file <none>
I20260620 11:46:15.134519 24646 fs_manager.cc:359] Metadata directory not provided
I20260620 11:46:15.134565 24646 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 11:46:15.134706 24646 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 11:46:15.135761 24646 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task1JRg7N/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781955746769051-7083-0/minicluster-data/ts-0/data/instance:
uuid: "88dda5569ce2408faa4c49eb9c977c6f"
format_stamp: "Formatted at 2026-06-20 11:46:15 on dist-test-slave-cjc6"
server_key: "5b08293d87ff2d335f15c4da50f5f1de"
server_key_iv: "cf7bce574e1ec0b7c7a8c5211bc9ad53"
server_key_version: "encryptionkey@0"
I20260620 11:46:15.136114 24646 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task1JRg7N/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781955746769051-7083-0/minicluster-data/ts-0/wal/instance:
uuid: "88dda5569ce2408faa4c49eb9c977c6f"
format_stamp: "Formatted at 2026-06-20 11:46:15 on dist-test-slave-cjc6"
server_key: "5b08293d87ff2d335f15c4da50f5f1de"
server_key_iv: "cf7bce574e1ec0b7c7a8c5211bc9ad53"
server_key_version: "encryptionkey@0"
I20260620 11:46:15.137429 24646 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260620 11:46:15.138255 24660 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 11:46:15.138407 24646 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.000s
I20260620 11:46:15.138497 24646 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task1JRg7N/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781955746769051-7083-0/minicluster-data/ts-0/data,/tmp/dist-test-task1JRg7N/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781955746769051-7083-0/minicluster-data/ts-0/wal
uuid: "88dda5569ce2408faa4c49eb9c977c6f"
format_stamp: "Formatted at 2026-06-20 11:46:15 on dist-test-slave-cjc6"
server_key: "5b08293d87ff2d335f15c4da50f5f1de"
server_key_iv: "cf7bce574e1ec0b7c7a8c5211bc9ad53"
server_key_version: "encryptionkey@0"
I20260620 11:46:15.138563 24646 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task1JRg7N/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781955746769051-7083-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-task1JRg7N/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781955746769051-7083-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-task1JRg7N/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781955746769051-7083-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 11:46:15.158517 24646 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 11:46:15.158838 24646 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 11:46:15.158990 24646 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 11:46:15.159217 24646 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 11:46:15.159543 24646 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 11:46:15.159576 24646 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 11:46:15.159597 24646 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 11:46:15.159642 24646 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 11:46:15.165558 24646 rpc_server.cc:307] RPC server started. Bound to: 127.6.234.193:41331
I20260620 11:46:15.165604 24773 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.6.234.193:41331 every 8 connection(s)
I20260620 11:46:15.165971 24646 server_base.cc:1166] Dumped server information to /tmp/dist-test-task1JRg7N/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781955746769051-7083-0/minicluster-data/ts-0/data/info.pb
I20260620 11:46:15.171271 24625 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 11:46:15.166684 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.6.234.193:60159 (local address 127.6.234.254:44741)
0620 11:46:15.166734 (+ 50us) server_negotiation.cc:206] Beginning negotiation
0620 11:46:15.166736 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 11:46:15.166945 (+ 209us) server_negotiation.cc:407] Connection header received
0620 11:46:15.167484 (+ 539us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 11:46:15.167485 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 11:46:15.167503 (+ 18us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 11:46:15.167563 (+ 60us) server_negotiation.cc:226] Negotiated authn=SASL
0620 11:46:15.168247 (+ 684us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 11:46:15.168756 (+ 509us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 11:46:15.169788 (+ 1032us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 11:46:15.170006 (+ 218us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 11:46:15.170095 (+ 89us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 11:46:15.170960 (+ 865us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 11:46:15.170961 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 11:46:15.170962 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 11:46:15.170967 (+ 5us) server_negotiation.cc:922] calling sasl_server_start()
0620 11:46:15.170982 (+ 15us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 11:46:15.170994 (+ 12us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 11:46:15.171047 (+ 53us) server_negotiation.cc:1035] Waiting for connection context
0620 11:46:15.171151 (+ 104us) server_negotiation.cc:299] Negotiation successful
0620 11:46:15.171192 (+ 41us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":23}
I20260620 11:46:15.171479 24776 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 11:46:15.166634 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.234.254:44741 (local address 127.6.234.193:60159)
0620 11:46:15.166819 (+ 185us) negotiation.cc:107] Waiting for socket to connect
0620 11:46:15.166854 (+ 35us) client_negotiation.cc:174] Beginning negotiation
0620 11:46:15.167374 (+ 520us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 11:46:15.167636 (+ 262us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 11:46:15.167640 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 11:46:15.167652 (+ 12us) client_negotiation.cc:189] Negotiated authn=SASL
0620 11:46:15.168150 (+ 498us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 11:46:15.168154 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 11:46:15.168856 (+ 702us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 11:46:15.168857 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 11:46:15.169652 (+ 795us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 11:46:15.169655 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 11:46:15.170106 (+ 451us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 11:46:15.170107 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 11:46:15.170250 (+ 143us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 11:46:15.170816 (+ 566us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0620 11:46:15.170817 (+ 1us) client_negotiation.cc:656] calling sasl_client_start()
0620 11:46:15.170845 (+ 28us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0620 11:46:15.170867 (+ 22us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 11:46:15.171058 (+ 191us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 11:46:15.171063 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 11:46:15.171066 (+ 3us) client_negotiation.cc:769] Sending connection context
0620 11:46:15.171153 (+ 87us) client_negotiation.cc:240] Negotiation successful
0620 11:46:15.171251 (+ 98us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":112,"thread_start_us":78,"threads_started":1}
I20260620 11:46:15.171669 7083 external_mini_cluster.cc:1388] Started /tmp/dist-test-task1JRg7N/build/release/bin/kudu as pid 24646
I20260620 11:46:15.171787 7083 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task1JRg7N/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781955746769051-7083-0/minicluster-data/ts-0/wal/instance
I20260620 11:46:15.172039 24774 heartbeater.cc:347] Connected to a master server at 127.6.234.254:44741
I20260620 11:46:15.172096 7083 external_mini_cluster.cc:1428] Setting key 71220317add50719753feef07adfdbf4
I20260620 11:46:15.172176 24774 heartbeater.cc:464] Registering TS with master...
I20260620 11:46:15.172361 24774 heartbeater.cc:511] Master 127.6.234.254:44741 requested a full tablet report, sending...
I20260620 11:46:15.172894 24586 ts_manager.cc:194] Registered new tserver with Master: 88dda5569ce2408faa4c49eb9c977c6f (127.6.234.193:41331)
I20260620 11:46:15.173100 7083 external_mini_cluster.cc:1326] Running /tmp/dist-test-task1JRg7N/build/release/bin/kudu
/tmp/dist-test-task1JRg7N/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task1JRg7N/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781955746769051-7083-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-task1JRg7N/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781955746769051-7083-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task1JRg7N/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781955746769051-7083-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-task1JRg7N/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781955746769051-7083-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.6.234.194:0
--local_ip_for_outbound_sockets=127.6.234.194
--webserver_interface=127.6.234.194
--webserver_port=0
--tserver_master_addrs=127.6.234.254:44741
--builtin_ntp_servers=127.6.234.212:34529
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
I20260620 11:46:15.173846 24586 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.6.234.193:60159
W20260620 11:46:15.253271 24777 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 11:46:15.253473 24777 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 11:46:15.253496 24777 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 11:46:15.254739 24777 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 11:46:15.254792 24777 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 11:46:15.254827 24777 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.6.234.194
I20260620 11:46:15.256137 24777 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.6.234.212:34529
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task1JRg7N/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781955746769051-7083-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-task1JRg7N/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781955746769051-7083-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.6.234.194:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task1JRg7N/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781955746769051-7083-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.6.234.194
--webserver_port=0
--tserver_master_addrs=127.6.234.254:44741
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.24777
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.6.234.194
--log_dir=/tmp/dist-test-task1JRg7N/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781955746769051-7083-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 20 Jun 2026 10:32:37 UTC on ccycloud.quasar-bgyofv.root.comops.site
build id 4457173
I20260620 11:46:15.256330 24777 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 11:46:15.256567 24777 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 11:46:15.261372 24783 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 11:46:15.261445 24782 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 11:46:15.261373 24785 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 11:46:15.261607 24777 server_base.cc:1034] running on GCE node
I20260620 11:46:15.261826 24777 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 11:46:15.262014 24777 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 11:46:15.263162 24777 hybrid_clock.cc:648] HybridClock initialized: now 1781955975263151 us; error 28 us; skew 500 ppm
I20260620 11:46:15.267586 24777 webserver.cc:460] Webserver started at http://127.6.234.194:38059/ using document root <none> and password file <none>
I20260620 11:46:15.267820 24777 fs_manager.cc:359] Metadata directory not provided
I20260620 11:46:15.267876 24777 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 11:46:15.268054 24777 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 11:46:15.269099 24777 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task1JRg7N/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781955746769051-7083-0/minicluster-data/ts-1/data/instance:
uuid: "ea80692d30704f638a4646edb9973a82"
format_stamp: "Formatted at 2026-06-20 11:46:15 on dist-test-slave-cjc6"
server_key: "77951b227724f37d7a480894e360019c"
server_key_iv: "c3521ca971849057387c1d6f22238463"
server_key_version: "encryptionkey@0"
I20260620 11:46:15.269446 24777 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task1JRg7N/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781955746769051-7083-0/minicluster-data/ts-1/wal/instance:
uuid: "ea80692d30704f638a4646edb9973a82"
format_stamp: "Formatted at 2026-06-20 11:46:15 on dist-test-slave-cjc6"
server_key: "77951b227724f37d7a480894e360019c"
server_key_iv: "c3521ca971849057387c1d6f22238463"
server_key_version: "encryptionkey@0"
I20260620 11:46:15.270777 24777 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.003s
I20260620 11:46:15.271505 24791 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 11:46:15.271692 24777 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.001s
I20260620 11:46:15.271780 24777 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task1JRg7N/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781955746769051-7083-0/minicluster-data/ts-1/data,/tmp/dist-test-task1JRg7N/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781955746769051-7083-0/minicluster-data/ts-1/wal
uuid: "ea80692d30704f638a4646edb9973a82"
format_stamp: "Formatted at 2026-06-20 11:46:15 on dist-test-slave-cjc6"
server_key: "77951b227724f37d7a480894e360019c"
server_key_iv: "c3521ca971849057387c1d6f22238463"
server_key_version: "encryptionkey@0"
I20260620 11:46:15.271857 24777 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task1JRg7N/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781955746769051-7083-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-task1JRg7N/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781955746769051-7083-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-task1JRg7N/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781955746769051-7083-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 11:46:15.291376 24777 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 11:46:15.291702 24777 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 11:46:15.291847 24777 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 11:46:15.292106 24777 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 11:46:15.292440 24777 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 11:46:15.292495 24777 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 11:46:15.292527 24777 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 11:46:15.292558 24777 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 11:46:15.298517 24777 rpc_server.cc:307] RPC server started. Bound to: 127.6.234.194:42983
I20260620 11:46:15.298564 24904 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.6.234.194:42983 every 8 connection(s)
I20260620 11:46:15.298908 24777 server_base.cc:1166] Dumped server information to /tmp/dist-test-task1JRg7N/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781955746769051-7083-0/minicluster-data/ts-1/data/info.pb
I20260620 11:46:15.303891 24625 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 11:46:15.299701 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.6.234.194:34733 (local address 127.6.234.254:44741)
0620 11:46:15.299747 (+ 46us) server_negotiation.cc:206] Beginning negotiation
0620 11:46:15.299749 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 11:46:15.299976 (+ 227us) server_negotiation.cc:407] Connection header received
0620 11:46:15.300458 (+ 482us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 11:46:15.300459 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 11:46:15.300470 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 11:46:15.300517 (+ 47us) server_negotiation.cc:226] Negotiated authn=SASL
0620 11:46:15.301159 (+ 642us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 11:46:15.301635 (+ 476us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 11:46:15.302627 (+ 992us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 11:46:15.302806 (+ 179us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 11:46:15.302858 (+ 52us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 11:46:15.303695 (+ 837us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 11:46:15.303696 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 11:46:15.303696 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 11:46:15.303701 (+ 5us) server_negotiation.cc:922] calling sasl_server_start()
0620 11:46:15.303711 (+ 10us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 11:46:15.303714 (+ 3us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 11:46:15.303751 (+ 37us) server_negotiation.cc:1035] Waiting for connection context
0620 11:46:15.303814 (+ 63us) server_negotiation.cc:299] Negotiation successful
0620 11:46:15.303832 (+ 18us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":20}
I20260620 11:46:15.304127 24907 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 11:46:15.299688 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.234.254:44741 (local address 127.6.234.194:34733)
0620 11:46:15.299852 (+ 164us) negotiation.cc:107] Waiting for socket to connect
0620 11:46:15.299884 (+ 32us) client_negotiation.cc:174] Beginning negotiation
0620 11:46:15.300372 (+ 488us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 11:46:15.300579 (+ 207us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 11:46:15.300582 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 11:46:15.300591 (+ 9us) client_negotiation.cc:189] Negotiated authn=SASL
0620 11:46:15.301090 (+ 499us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 11:46:15.301094 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 11:46:15.301750 (+ 656us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 11:46:15.301751 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 11:46:15.302555 (+ 804us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 11:46:15.302557 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 11:46:15.302880 (+ 323us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 11:46:15.302881 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 11:46:15.302983 (+ 102us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 11:46:15.303575 (+ 592us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0620 11:46:15.303575 (+ 0us) client_negotiation.cc:656] calling sasl_client_start()
0620 11:46:15.303603 (+ 28us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0620 11:46:15.303625 (+ 22us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 11:46:15.303753 (+ 128us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 11:46:15.303755 (+ 2us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 11:46:15.303758 (+ 3us) client_negotiation.cc:769] Sending connection context
0620 11:46:15.303803 (+ 45us) client_negotiation.cc:240] Negotiation successful
0620 11:46:15.303876 (+ 73us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":106,"thread_start_us":79,"threads_started":1}
I20260620 11:46:15.304683 24905 heartbeater.cc:347] Connected to a master server at 127.6.234.254:44741
I20260620 11:46:15.304781 24905 heartbeater.cc:464] Registering TS with master...
I20260620 11:46:15.304971 24905 heartbeater.cc:511] Master 127.6.234.254:44741 requested a full tablet report, sending...
I20260620 11:46:15.305362 24586 ts_manager.cc:194] Registered new tserver with Master: ea80692d30704f638a4646edb9973a82 (127.6.234.194:42983)
I20260620 11:46:15.305784 24586 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.6.234.194:34733
I20260620 11:46:15.308127 7083 external_mini_cluster.cc:1388] Started /tmp/dist-test-task1JRg7N/build/release/bin/kudu as pid 24777
I20260620 11:46:15.308225 7083 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task1JRg7N/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781955746769051-7083-0/minicluster-data/ts-1/wal/instance
I20260620 11:46:15.308380 7083 external_mini_cluster.cc:1428] Setting key 5dbf31085d0ed957506222bec94a2bb6
I20260620 11:46:15.309360 7083 external_mini_cluster.cc:1326] Running /tmp/dist-test-task1JRg7N/build/release/bin/kudu
/tmp/dist-test-task1JRg7N/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task1JRg7N/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781955746769051-7083-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-task1JRg7N/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781955746769051-7083-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task1JRg7N/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781955746769051-7083-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-task1JRg7N/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781955746769051-7083-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.6.234.195:0
--local_ip_for_outbound_sockets=127.6.234.195
--webserver_interface=127.6.234.195
--webserver_port=0
--tserver_master_addrs=127.6.234.254:44741
--builtin_ntp_servers=127.6.234.212:34529
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
W20260620 11:46:15.390555 24908 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 11:46:15.390753 24908 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 11:46:15.390774 24908 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 11:46:15.391899 24908 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 11:46:15.391944 24908 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 11:46:15.391969 24908 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.6.234.195
I20260620 11:46:15.393388 24908 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.6.234.212:34529
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task1JRg7N/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781955746769051-7083-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-task1JRg7N/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781955746769051-7083-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.6.234.195:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task1JRg7N/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781955746769051-7083-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.6.234.195
--webserver_port=0
--tserver_master_addrs=127.6.234.254:44741
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.24908
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.6.234.195
--log_dir=/tmp/dist-test-task1JRg7N/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781955746769051-7083-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 20 Jun 2026 10:32:37 UTC on ccycloud.quasar-bgyofv.root.comops.site
build id 4457173
I20260620 11:46:15.393687 24908 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 11:46:15.393998 24908 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 11:46:15.398799 24914 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 11:46:15.398790 24916 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 11:46:15.398988 24908 server_base.cc:1034] running on GCE node
W20260620 11:46:15.399104 24913 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 11:46:15.399322 24908 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 11:46:15.399574 24908 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 11:46:15.400736 24908 hybrid_clock.cc:648] HybridClock initialized: now 1781955975400726 us; error 57 us; skew 500 ppm
I20260620 11:46:15.402055 24908 webserver.cc:460] Webserver started at http://127.6.234.195:40265/ using document root <none> and password file <none>
I20260620 11:46:15.402308 24908 fs_manager.cc:359] Metadata directory not provided
I20260620 11:46:15.402379 24908 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 11:46:15.402535 24908 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 11:46:15.403548 24908 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task1JRg7N/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781955746769051-7083-0/minicluster-data/ts-2/data/instance:
uuid: "55d862e64c3d4bffab828ccb15c5f88d"
format_stamp: "Formatted at 2026-06-20 11:46:15 on dist-test-slave-cjc6"
server_key: "7ff27b685a836c58e9f5a4ae8ce5c811"
server_key_iv: "708736966def8d84e9c0c0a00ef7a5ba"
server_key_version: "encryptionkey@0"
I20260620 11:46:15.403908 24908 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task1JRg7N/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781955746769051-7083-0/minicluster-data/ts-2/wal/instance:
uuid: "55d862e64c3d4bffab828ccb15c5f88d"
format_stamp: "Formatted at 2026-06-20 11:46:15 on dist-test-slave-cjc6"
server_key: "7ff27b685a836c58e9f5a4ae8ce5c811"
server_key_iv: "708736966def8d84e9c0c0a00ef7a5ba"
server_key_version: "encryptionkey@0"
I20260620 11:46:15.405167 24908 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260620 11:46:15.405894 24922 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 11:46:15.406071 24908 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260620 11:46:15.406172 24908 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task1JRg7N/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781955746769051-7083-0/minicluster-data/ts-2/data,/tmp/dist-test-task1JRg7N/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781955746769051-7083-0/minicluster-data/ts-2/wal
uuid: "55d862e64c3d4bffab828ccb15c5f88d"
format_stamp: "Formatted at 2026-06-20 11:46:15 on dist-test-slave-cjc6"
server_key: "7ff27b685a836c58e9f5a4ae8ce5c811"
server_key_iv: "708736966def8d84e9c0c0a00ef7a5ba"
server_key_version: "encryptionkey@0"
I20260620 11:46:15.406250 24908 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task1JRg7N/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781955746769051-7083-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-task1JRg7N/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781955746769051-7083-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-task1JRg7N/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781955746769051-7083-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 11:46:15.433776 24908 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 11:46:15.434119 24908 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 11:46:15.434283 24908 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 11:46:15.434523 24908 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 11:46:15.434856 24908 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 11:46:15.434913 24908 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 11:46:15.434966 24908 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 11:46:15.434993 24908 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 11:46:15.441093 24908 rpc_server.cc:307] RPC server started. Bound to: 127.6.234.195:43621
I20260620 11:46:15.441149 25035 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.6.234.195:43621 every 8 connection(s)
I20260620 11:46:15.441488 24908 server_base.cc:1166] Dumped server information to /tmp/dist-test-task1JRg7N/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781955746769051-7083-0/minicluster-data/ts-2/data/info.pb
I20260620 11:46:15.444475 7083 external_mini_cluster.cc:1388] Started /tmp/dist-test-task1JRg7N/build/release/bin/kudu as pid 24908
I20260620 11:46:15.444584 7083 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task1JRg7N/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781955746769051-7083-0/minicluster-data/ts-2/wal/instance
I20260620 11:46:15.444795 7083 external_mini_cluster.cc:1428] Setting key 55d8514270a94672c3df8e84a6cfe23b
I20260620 11:46:15.447150 24625 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 11:46:15.442196 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.6.234.195:34661 (local address 127.6.234.254:44741)
0620 11:46:15.442279 (+ 83us) server_negotiation.cc:206] Beginning negotiation
0620 11:46:15.442281 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 11:46:15.442522 (+ 241us) server_negotiation.cc:407] Connection header received
0620 11:46:15.443029 (+ 507us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 11:46:15.443030 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 11:46:15.443041 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 11:46:15.443093 (+ 52us) server_negotiation.cc:226] Negotiated authn=SASL
0620 11:46:15.443774 (+ 681us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 11:46:15.444258 (+ 484us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 11:46:15.445688 (+ 1430us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 11:46:15.445982 (+ 294us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 11:46:15.446040 (+ 58us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 11:46:15.446943 (+ 903us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 11:46:15.446944 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 11:46:15.446944 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 11:46:15.446948 (+ 4us) server_negotiation.cc:922] calling sasl_server_start()
0620 11:46:15.446958 (+ 10us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 11:46:15.446961 (+ 3us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 11:46:15.447001 (+ 40us) server_negotiation.cc:1035] Waiting for connection context
0620 11:46:15.447054 (+ 53us) server_negotiation.cc:299] Negotiation successful
0620 11:46:15.447072 (+ 18us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":47}
I20260620 11:46:15.447420 25038 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 11:46:15.442170 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.234.254:44741 (local address 127.6.234.195:34661)
0620 11:46:15.442389 (+ 219us) negotiation.cc:107] Waiting for socket to connect
0620 11:46:15.442425 (+ 36us) client_negotiation.cc:174] Beginning negotiation
0620 11:46:15.442943 (+ 518us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 11:46:15.443153 (+ 210us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 11:46:15.443157 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 11:46:15.443165 (+ 8us) client_negotiation.cc:189] Negotiated authn=SASL
0620 11:46:15.443699 (+ 534us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 11:46:15.443703 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 11:46:15.444370 (+ 667us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 11:46:15.444371 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 11:46:15.445583 (+ 1212us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 11:46:15.445586 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 11:46:15.446062 (+ 476us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 11:46:15.446063 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 11:46:15.446185 (+ 122us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 11:46:15.446814 (+ 629us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0620 11:46:15.446814 (+ 0us) client_negotiation.cc:656] calling sasl_client_start()
0620 11:46:15.446844 (+ 30us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0620 11:46:15.446869 (+ 25us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 11:46:15.447001 (+ 132us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 11:46:15.447004 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 11:46:15.447006 (+ 2us) client_negotiation.cc:769] Sending connection context
0620 11:46:15.447056 (+ 50us) client_negotiation.cc:240] Negotiation successful
0620 11:46:15.447153 (+ 97us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":159,"thread_start_us":111,"threads_started":1}
I20260620 11:46:15.447878 25036 heartbeater.cc:347] Connected to a master server at 127.6.234.254:44741
I20260620 11:46:15.447971 25036 heartbeater.cc:464] Registering TS with master...
I20260620 11:46:15.448170 25036 heartbeater.cc:511] Master 127.6.234.254:44741 requested a full tablet report, sending...
I20260620 11:46:15.448518 24586 ts_manager.cc:194] Registered new tserver with Master: 55d862e64c3d4bffab828ccb15c5f88d (127.6.234.195:43621)
I20260620 11:46:15.448889 24586 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.6.234.195:34661
I20260620 11:46:15.456601 7083 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260620 11:46:15.460529 24625 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 11:46:15.458019 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59668 (local address 127.6.234.254:44741)
0620 11:46:15.458087 (+ 68us) server_negotiation.cc:206] Beginning negotiation
0620 11:46:15.458089 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 11:46:15.458188 (+ 99us) server_negotiation.cc:407] Connection header received
0620 11:46:15.458215 (+ 27us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 11:46:15.458215 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 11:46:15.458224 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 11:46:15.458272 (+ 48us) server_negotiation.cc:226] Negotiated authn=SASL
0620 11:46:15.458400 (+ 128us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 11:46:15.458855 (+ 455us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 11:46:15.459544 (+ 689us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 11:46:15.459719 (+ 175us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 11:46:15.459766 (+ 47us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 11:46:15.460351 (+ 585us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 11:46:15.460352 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 11:46:15.460353 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 11:46:15.460357 (+ 4us) server_negotiation.cc:922] calling sasl_server_start()
0620 11:46:15.460369 (+ 12us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 11:46:15.460373 (+ 4us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 11:46:15.460415 (+ 42us) server_negotiation.cc:1035] Waiting for connection context
0620 11:46:15.460459 (+ 44us) server_negotiation.cc:299] Negotiation successful
0620 11:46:15.460477 (+ 18us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":41}
I20260620 11:46:15.462212 24586 catalog_manager.cc:1908] Servicing CreateTable request from {username='slave'} at 127.0.0.1:59668:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260620 11:46:15.463737 24586 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260620 11:46:15.470254 25044 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 11:46:15.465911 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.234.194:42983 (local address 127.0.0.1:40674)
0620 11:46:15.466277 (+ 366us) negotiation.cc:107] Waiting for socket to connect
0620 11:46:15.466316 (+ 39us) client_negotiation.cc:174] Beginning negotiation
0620 11:46:15.466372 (+ 56us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 11:46:15.466563 (+ 191us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 11:46:15.466566 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 11:46:15.466567 (+ 1us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 11:46:15.466640 (+ 73us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 11:46:15.466642 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 11:46:15.467984 (+ 1342us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 11:46:15.467985 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 11:46:15.469060 (+ 1075us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 11:46:15.469062 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 11:46:15.469933 (+ 871us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 11:46:15.469935 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 11:46:15.470014 (+ 79us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 11:46:15.470028 (+ 14us) client_negotiation.cc:769] Sending connection context
0620 11:46:15.470111 (+ 83us) client_negotiation.cc:240] Negotiation successful
0620 11:46:15.470165 (+ 54us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":330,"mutex_wait_us":35,"thread_start_us":108,"threads_started":1}
I20260620 11:46:15.470284 25046 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 11:46:15.466024 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40674 (local address 127.6.234.194:42983)
0620 11:46:15.466380 (+ 356us) server_negotiation.cc:206] Beginning negotiation
0620 11:46:15.466385 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 11:46:15.466397 (+ 12us) server_negotiation.cc:407] Connection header received
0620 11:46:15.466441 (+ 44us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 11:46:15.466443 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 11:46:15.466459 (+ 16us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 11:46:15.466565 (+ 106us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 11:46:15.466725 (+ 160us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 11:46:15.467892 (+ 1167us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 11:46:15.469144 (+ 1252us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 11:46:15.469840 (+ 696us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 11:46:15.469916 (+ 76us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 11:46:15.470007 (+ 91us) server_negotiation.cc:1035] Waiting for connection context
0620 11:46:15.470139 (+ 132us) server_negotiation.cc:299] Negotiation successful
0620 11:46:15.470184 (+ 45us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":315,"thread_start_us":100,"threads_started":1}
I20260620 11:46:15.470492 25047 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 11:46:15.466090 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.234.195:43621 (local address 127.0.0.1:57984)
0620 11:46:15.466945 (+ 855us) negotiation.cc:107] Waiting for socket to connect
0620 11:46:15.466954 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0620 11:46:15.467006 (+ 52us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 11:46:15.467194 (+ 188us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 11:46:15.467195 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 11:46:15.467197 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 11:46:15.467239 (+ 42us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 11:46:15.467241 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 11:46:15.468130 (+ 889us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 11:46:15.468130 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 11:46:15.469336 (+ 1206us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 11:46:15.469339 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 11:46:15.470234 (+ 895us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 11:46:15.470234 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 11:46:15.470295 (+ 61us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 11:46:15.470299 (+ 4us) client_negotiation.cc:769] Sending connection context
0620 11:46:15.470355 (+ 56us) client_negotiation.cc:240] Negotiation successful
0620 11:46:15.470399 (+ 44us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":829,"mutex_wait_us":36,"thread_start_us":69,"threads_started":1}
I20260620 11:46:15.470752 25049 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 11:46:15.466179 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57984 (local address 127.6.234.195:43621)
0620 11:46:15.466356 (+ 177us) server_negotiation.cc:206] Beginning negotiation
0620 11:46:15.466361 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 11:46:15.467004 (+ 643us) server_negotiation.cc:407] Connection header received
0620 11:46:15.467081 (+ 77us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 11:46:15.467083 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 11:46:15.467098 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 11:46:15.467197 (+ 99us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 11:46:15.467294 (+ 97us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 11:46:15.468027 (+ 733us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 11:46:15.469445 (+ 1418us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 11:46:15.470058 (+ 613us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 11:46:15.470119 (+ 61us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 11:46:15.470201 (+ 82us) server_negotiation.cc:1035] Waiting for connection context
0620 11:46:15.470638 (+ 437us) server_negotiation.cc:299] Negotiation successful
0620 11:46:15.470674 (+ 36us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":134,"thread_start_us":80,"threads_started":1}
I20260620 11:46:15.470829 25045 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 11:46:15.465867 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.234.193:41331 (local address 127.0.0.1:55744)
0620 11:46:15.466238 (+ 371us) negotiation.cc:107] Waiting for socket to connect
0620 11:46:15.466294 (+ 56us) client_negotiation.cc:174] Beginning negotiation
0620 11:46:15.466367 (+ 73us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 11:46:15.466851 (+ 484us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 11:46:15.466852 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 11:46:15.466854 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 11:46:15.466890 (+ 36us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 11:46:15.466891 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 11:46:15.468096 (+ 1205us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 11:46:15.468097 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 11:46:15.469342 (+ 1245us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 11:46:15.469344 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 11:46:15.470592 (+ 1248us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 11:46:15.470593 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 11:46:15.470661 (+ 68us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 11:46:15.470665 (+ 4us) client_negotiation.cc:769] Sending connection context
0620 11:46:15.470715 (+ 50us) client_negotiation.cc:240] Negotiation successful
0620 11:46:15.470756 (+ 41us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":331,"mutex_wait_us":67,"thread_start_us":75,"threads_started":1}
I20260620 11:46:15.471093 25048 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 11:46:15.466054 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55744 (local address 127.6.234.193:41331)
0620 11:46:15.466698 (+ 644us) server_negotiation.cc:206] Beginning negotiation
0620 11:46:15.466702 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 11:46:15.466712 (+ 10us) server_negotiation.cc:407] Connection header received
0620 11:46:15.466749 (+ 37us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 11:46:15.466750 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 11:46:15.466764 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 11:46:15.466840 (+ 76us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 11:46:15.466975 (+ 135us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 11:46:15.468014 (+ 1039us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 11:46:15.469422 (+ 1408us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 11:46:15.470188 (+ 766us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 11:46:15.470255 (+ 67us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 11:46:15.470553 (+ 298us) server_negotiation.cc:1035] Waiting for connection context
0620 11:46:15.470941 (+ 388us) server_negotiation.cc:299] Negotiation successful
0620 11:46:15.470999 (+ 58us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":599,"thread_start_us":108,"threads_started":1}
I20260620 11:46:15.471169 24839 tablet_service.cc:1467] Processing CreateTablet for tablet f194d5eef3fc4d7a97398272db01c8aa (DEFAULT_TABLE table=test-table [id=a29909ce88314fe4af2658f4135ce565]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 11:46:15.471453 24839 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f194d5eef3fc4d7a97398272db01c8aa. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 11:46:15.471632 24708 tablet_service.cc:1467] Processing CreateTablet for tablet f194d5eef3fc4d7a97398272db01c8aa (DEFAULT_TABLE table=test-table [id=a29909ce88314fe4af2658f4135ce565]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 11:46:15.471873 24708 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f194d5eef3fc4d7a97398272db01c8aa. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 11:46:15.473937 25050 tablet_bootstrap.cc:492] T f194d5eef3fc4d7a97398272db01c8aa P ea80692d30704f638a4646edb9973a82: Bootstrap starting.
I20260620 11:46:15.474633 25050 tablet_bootstrap.cc:654] T f194d5eef3fc4d7a97398272db01c8aa P ea80692d30704f638a4646edb9973a82: Neither blocks nor log segments found. Creating new log.
I20260620 11:46:15.474658 24970 tablet_service.cc:1467] Processing CreateTablet for tablet f194d5eef3fc4d7a97398272db01c8aa (DEFAULT_TABLE table=test-table [id=a29909ce88314fe4af2658f4135ce565]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 11:46:15.474900 24970 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f194d5eef3fc4d7a97398272db01c8aa. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 11:46:15.474987 25050 log.cc:824] T f194d5eef3fc4d7a97398272db01c8aa P ea80692d30704f638a4646edb9973a82: Log is configured to *not* fsync() on all Append() calls
I20260620 11:46:15.475749 25050 tablet_bootstrap.cc:492] T f194d5eef3fc4d7a97398272db01c8aa P ea80692d30704f638a4646edb9973a82: No bootstrap required, opened a new log
I20260620 11:46:15.475829 25050 ts_tablet_manager.cc:1397] T f194d5eef3fc4d7a97398272db01c8aa P ea80692d30704f638a4646edb9973a82: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260620 11:46:15.477383 25053 tablet_bootstrap.cc:492] T f194d5eef3fc4d7a97398272db01c8aa P 55d862e64c3d4bffab828ccb15c5f88d: Bootstrap starting.
I20260620 11:46:15.477628 25051 tablet_bootstrap.cc:492] T f194d5eef3fc4d7a97398272db01c8aa P 88dda5569ce2408faa4c49eb9c977c6f: Bootstrap starting.
I20260620 11:46:15.477826 25050 raft_consensus.cc:348] T f194d5eef3fc4d7a97398272db01c8aa P ea80692d30704f638a4646edb9973a82 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88dda5569ce2408faa4c49eb9c977c6f" member_type: VOTER last_known_addr { host: "127.6.234.193" port: 41331 } } peers { permanent_uuid: "ea80692d30704f638a4646edb9973a82" member_type: VOTER last_known_addr { host: "127.6.234.194" port: 42983 } } peers { permanent_uuid: "55d862e64c3d4bffab828ccb15c5f88d" member_type: VOTER last_known_addr { host: "127.6.234.195" port: 43621 } }
I20260620 11:46:15.477980 25050 raft_consensus.cc:374] T f194d5eef3fc4d7a97398272db01c8aa P ea80692d30704f638a4646edb9973a82 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 11:46:15.478039 25050 raft_consensus.cc:729] T f194d5eef3fc4d7a97398272db01c8aa P ea80692d30704f638a4646edb9973a82 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ea80692d30704f638a4646edb9973a82, State: Initialized, Role: FOLLOWER
I20260620 11:46:15.478159 25053 tablet_bootstrap.cc:654] T f194d5eef3fc4d7a97398272db01c8aa P 55d862e64c3d4bffab828ccb15c5f88d: Neither blocks nor log segments found. Creating new log.
I20260620 11:46:15.478163 25050 consensus_queue.cc:260] T f194d5eef3fc4d7a97398272db01c8aa P ea80692d30704f638a4646edb9973a82 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88dda5569ce2408faa4c49eb9c977c6f" member_type: VOTER last_known_addr { host: "127.6.234.193" port: 41331 } } peers { permanent_uuid: "ea80692d30704f638a4646edb9973a82" member_type: VOTER last_known_addr { host: "127.6.234.194" port: 42983 } } peers { permanent_uuid: "55d862e64c3d4bffab828ccb15c5f88d" member_type: VOTER last_known_addr { host: "127.6.234.195" port: 43621 } }
I20260620 11:46:15.478312 25051 tablet_bootstrap.cc:654] T f194d5eef3fc4d7a97398272db01c8aa P 88dda5569ce2408faa4c49eb9c977c6f: Neither blocks nor log segments found. Creating new log.
I20260620 11:46:15.478492 25050 ts_tablet_manager.cc:1428] T f194d5eef3fc4d7a97398272db01c8aa P ea80692d30704f638a4646edb9973a82: Time spent starting tablet: real 0.003s user 0.003s sys 0.000s
I20260620 11:46:15.478498 25053 log.cc:824] T f194d5eef3fc4d7a97398272db01c8aa P 55d862e64c3d4bffab828ccb15c5f88d: Log is configured to *not* fsync() on all Append() calls
I20260620 11:46:15.478638 24905 heartbeater.cc:503] Master 127.6.234.254:44741 was elected leader, sending a full tablet report...
I20260620 11:46:15.478675 25051 log.cc:824] T f194d5eef3fc4d7a97398272db01c8aa P 88dda5569ce2408faa4c49eb9c977c6f: Log is configured to *not* fsync() on all Append() calls
I20260620 11:46:15.479231 25053 tablet_bootstrap.cc:492] T f194d5eef3fc4d7a97398272db01c8aa P 55d862e64c3d4bffab828ccb15c5f88d: No bootstrap required, opened a new log
I20260620 11:46:15.479305 25053 ts_tablet_manager.cc:1397] T f194d5eef3fc4d7a97398272db01c8aa P 55d862e64c3d4bffab828ccb15c5f88d: Time spent bootstrapping tablet: real 0.002s user 0.000s sys 0.001s
I20260620 11:46:15.479389 25051 tablet_bootstrap.cc:492] T f194d5eef3fc4d7a97398272db01c8aa P 88dda5569ce2408faa4c49eb9c977c6f: No bootstrap required, opened a new log
I20260620 11:46:15.479456 25051 ts_tablet_manager.cc:1397] T f194d5eef3fc4d7a97398272db01c8aa P 88dda5569ce2408faa4c49eb9c977c6f: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260620 11:46:15.481891 25053 raft_consensus.cc:348] T f194d5eef3fc4d7a97398272db01c8aa P 55d862e64c3d4bffab828ccb15c5f88d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88dda5569ce2408faa4c49eb9c977c6f" member_type: VOTER last_known_addr { host: "127.6.234.193" port: 41331 } } peers { permanent_uuid: "ea80692d30704f638a4646edb9973a82" member_type: VOTER last_known_addr { host: "127.6.234.194" port: 42983 } } peers { permanent_uuid: "55d862e64c3d4bffab828ccb15c5f88d" member_type: VOTER last_known_addr { host: "127.6.234.195" port: 43621 } }
I20260620 11:46:15.482059 25053 raft_consensus.cc:374] T f194d5eef3fc4d7a97398272db01c8aa P 55d862e64c3d4bffab828ccb15c5f88d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 11:46:15.482095 25053 raft_consensus.cc:729] T f194d5eef3fc4d7a97398272db01c8aa P 55d862e64c3d4bffab828ccb15c5f88d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 55d862e64c3d4bffab828ccb15c5f88d, State: Initialized, Role: FOLLOWER
I20260620 11:46:15.482026 25051 raft_consensus.cc:348] T f194d5eef3fc4d7a97398272db01c8aa P 88dda5569ce2408faa4c49eb9c977c6f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88dda5569ce2408faa4c49eb9c977c6f" member_type: VOTER last_known_addr { host: "127.6.234.193" port: 41331 } } peers { permanent_uuid: "ea80692d30704f638a4646edb9973a82" member_type: VOTER last_known_addr { host: "127.6.234.194" port: 42983 } } peers { permanent_uuid: "55d862e64c3d4bffab828ccb15c5f88d" member_type: VOTER last_known_addr { host: "127.6.234.195" port: 43621 } }
I20260620 11:46:15.482164 25051 raft_consensus.cc:374] T f194d5eef3fc4d7a97398272db01c8aa P 88dda5569ce2408faa4c49eb9c977c6f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 11:46:15.482204 25051 raft_consensus.cc:729] T f194d5eef3fc4d7a97398272db01c8aa P 88dda5569ce2408faa4c49eb9c977c6f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 88dda5569ce2408faa4c49eb9c977c6f, State: Initialized, Role: FOLLOWER
I20260620 11:46:15.482211 25053 consensus_queue.cc:260] T f194d5eef3fc4d7a97398272db01c8aa P 55d862e64c3d4bffab828ccb15c5f88d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88dda5569ce2408faa4c49eb9c977c6f" member_type: VOTER last_known_addr { host: "127.6.234.193" port: 41331 } } peers { permanent_uuid: "ea80692d30704f638a4646edb9973a82" member_type: VOTER last_known_addr { host: "127.6.234.194" port: 42983 } } peers { permanent_uuid: "55d862e64c3d4bffab828ccb15c5f88d" member_type: VOTER last_known_addr { host: "127.6.234.195" port: 43621 } }
I20260620 11:46:15.482306 25051 consensus_queue.cc:260] T f194d5eef3fc4d7a97398272db01c8aa P 88dda5569ce2408faa4c49eb9c977c6f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88dda5569ce2408faa4c49eb9c977c6f" member_type: VOTER last_known_addr { host: "127.6.234.193" port: 41331 } } peers { permanent_uuid: "ea80692d30704f638a4646edb9973a82" member_type: VOTER last_known_addr { host: "127.6.234.194" port: 42983 } } peers { permanent_uuid: "55d862e64c3d4bffab828ccb15c5f88d" member_type: VOTER last_known_addr { host: "127.6.234.195" port: 43621 } }
I20260620 11:46:15.482483 25053 ts_tablet_manager.cc:1428] T f194d5eef3fc4d7a97398272db01c8aa P 55d862e64c3d4bffab828ccb15c5f88d: Time spent starting tablet: real 0.003s user 0.002s sys 0.001s
I20260620 11:46:15.482551 25051 ts_tablet_manager.cc:1428] T f194d5eef3fc4d7a97398272db01c8aa P 88dda5569ce2408faa4c49eb9c977c6f: Time spent starting tablet: real 0.003s user 0.003s sys 0.000s
I20260620 11:46:15.482522 25036 heartbeater.cc:503] Master 127.6.234.254:44741 was elected leader, sending a full tablet report...
I20260620 11:46:15.482611 24774 heartbeater.cc:503] Master 127.6.234.254:44741 was elected leader, sending a full tablet report...
W20260620 11:46:15.549521 24906 tablet.cc:2369] T f194d5eef3fc4d7a97398272db01c8aa P ea80692d30704f638a4646edb9973a82: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260620 11:46:15.666639 24775 tablet.cc:2369] T f194d5eef3fc4d7a97398272db01c8aa P 88dda5569ce2408faa4c49eb9c977c6f: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260620 11:46:15.669297 25057 raft_consensus.cc:482] T f194d5eef3fc4d7a97398272db01c8aa P 55d862e64c3d4bffab828ccb15c5f88d [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260620 11:46:15.669422 25057 raft_consensus.cc:504] T f194d5eef3fc4d7a97398272db01c8aa P 55d862e64c3d4bffab828ccb15c5f88d [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88dda5569ce2408faa4c49eb9c977c6f" member_type: VOTER last_known_addr { host: "127.6.234.193" port: 41331 } } peers { permanent_uuid: "ea80692d30704f638a4646edb9973a82" member_type: VOTER last_known_addr { host: "127.6.234.194" port: 42983 } } peers { permanent_uuid: "55d862e64c3d4bffab828ccb15c5f88d" member_type: VOTER last_known_addr { host: "127.6.234.195" port: 43621 } }
I20260620 11:46:15.669869 25057 leader_election.cc:288] T f194d5eef3fc4d7a97398272db01c8aa P 55d862e64c3d4bffab828ccb15c5f88d [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 88dda5569ce2408faa4c49eb9c977c6f (127.6.234.193:41331), ea80692d30704f638a4646edb9973a82 (127.6.234.194:42983)
I20260620 11:46:15.672938 25046 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 11:46:15.670166 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.6.234.195:52147 (local address 127.6.234.194:42983)
0620 11:46:15.670214 (+ 48us) server_negotiation.cc:206] Beginning negotiation
0620 11:46:15.670216 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 11:46:15.670227 (+ 11us) server_negotiation.cc:407] Connection header received
0620 11:46:15.670259 (+ 32us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 11:46:15.670260 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 11:46:15.670273 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 11:46:15.670359 (+ 86us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 11:46:15.670553 (+ 194us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 11:46:15.671080 (+ 527us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 11:46:15.672026 (+ 946us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 11:46:15.672583 (+ 557us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 11:46:15.672659 (+ 76us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 11:46:15.672669 (+ 10us) server_negotiation.cc:1035] Waiting for connection context
0620 11:46:15.672817 (+ 148us) server_negotiation.cc:299] Negotiation successful
0620 11:46:15.672868 (+ 51us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":25}
I20260620 11:46:15.672919 25038 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 11:46:15.670083 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.234.194:42983 (local address 127.6.234.195:52147)
0620 11:46:15.670139 (+ 56us) negotiation.cc:107] Waiting for socket to connect
0620 11:46:15.670152 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0620 11:46:15.670197 (+ 45us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 11:46:15.670391 (+ 194us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 11:46:15.670392 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 11:46:15.670394 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 11:46:15.670471 (+ 77us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 11:46:15.670473 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 11:46:15.671171 (+ 698us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 11:46:15.671172 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 11:46:15.671920 (+ 748us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 11:46:15.671922 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 11:46:15.672661 (+ 739us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 11:46:15.672662 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 11:46:15.672735 (+ 73us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 11:46:15.672738 (+ 3us) client_negotiation.cc:769] Sending connection context
0620 11:46:15.672793 (+ 55us) client_negotiation.cc:240] Negotiation successful
0620 11:46:15.672839 (+ 46us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":37}
I20260620 11:46:15.673017 25048 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 11:46:15.670170 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.6.234.195:40545 (local address 127.6.234.193:41331)
0620 11:46:15.670213 (+ 43us) server_negotiation.cc:206] Beginning negotiation
0620 11:46:15.670216 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0620 11:46:15.670406 (+ 190us) server_negotiation.cc:407] Connection header received
0620 11:46:15.670441 (+ 35us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 11:46:15.670443 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 11:46:15.670458 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 11:46:15.670530 (+ 72us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 11:46:15.670680 (+ 150us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 11:46:15.671152 (+ 472us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 11:46:15.672149 (+ 997us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 11:46:15.672669 (+ 520us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 11:46:15.672740 (+ 71us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 11:46:15.672746 (+ 6us) server_negotiation.cc:1035] Waiting for connection context
0620 11:46:15.672893 (+ 147us) server_negotiation.cc:299] Negotiation successful
0620 11:46:15.672935 (+ 42us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":21}
I20260620 11:46:15.673061 25059 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 11:46:15.670094 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.234.193:41331 (local address 127.6.234.195:40545)
0620 11:46:15.670321 (+ 227us) negotiation.cc:107] Waiting for socket to connect
0620 11:46:15.670332 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0620 11:46:15.670390 (+ 58us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 11:46:15.670538 (+ 148us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 11:46:15.670539 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 11:46:15.670541 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 11:46:15.670607 (+ 66us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 11:46:15.670608 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 11:46:15.671230 (+ 622us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 11:46:15.671230 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 11:46:15.672065 (+ 835us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 11:46:15.672067 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 11:46:15.672743 (+ 676us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 11:46:15.672745 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 11:46:15.672821 (+ 76us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 11:46:15.672823 (+ 2us) client_negotiation.cc:769] Sending connection context
0620 11:46:15.672873 (+ 50us) client_negotiation.cc:240] Negotiation successful
0620 11:46:15.672901 (+ 28us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":199,"mutex_wait_us":56,"thread_start_us":102,"threads_started":1}
I20260620 11:46:15.673377 24728 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f194d5eef3fc4d7a97398272db01c8aa" candidate_uuid: "55d862e64c3d4bffab828ccb15c5f88d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "88dda5569ce2408faa4c49eb9c977c6f" is_pre_election: true
I20260620 11:46:15.673434 24859 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f194d5eef3fc4d7a97398272db01c8aa" candidate_uuid: "55d862e64c3d4bffab828ccb15c5f88d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ea80692d30704f638a4646edb9973a82" is_pre_election: true
I20260620 11:46:15.673511 24728 raft_consensus.cc:2454] T f194d5eef3fc4d7a97398272db01c8aa P 88dda5569ce2408faa4c49eb9c977c6f [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 55d862e64c3d4bffab828ccb15c5f88d in term 0.
I20260620 11:46:15.673605 24859 raft_consensus.cc:2454] T f194d5eef3fc4d7a97398272db01c8aa P ea80692d30704f638a4646edb9973a82 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 55d862e64c3d4bffab828ccb15c5f88d in term 0.
I20260620 11:46:15.673784 24923 leader_election.cc:302] T f194d5eef3fc4d7a97398272db01c8aa P 55d862e64c3d4bffab828ccb15c5f88d [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 55d862e64c3d4bffab828ccb15c5f88d, 88dda5569ce2408faa4c49eb9c977c6f; no voters:
I20260620 11:46:15.673928 25057 raft_consensus.cc:2781] T f194d5eef3fc4d7a97398272db01c8aa P 55d862e64c3d4bffab828ccb15c5f88d [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260620 11:46:15.674005 25057 raft_consensus.cc:482] T f194d5eef3fc4d7a97398272db01c8aa P 55d862e64c3d4bffab828ccb15c5f88d [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260620 11:46:15.674041 25057 raft_consensus.cc:3037] T f194d5eef3fc4d7a97398272db01c8aa P 55d862e64c3d4bffab828ccb15c5f88d [term 0 FOLLOWER]: Advancing to term 1
I20260620 11:46:15.674715 25057 raft_consensus.cc:504] T f194d5eef3fc4d7a97398272db01c8aa P 55d862e64c3d4bffab828ccb15c5f88d [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88dda5569ce2408faa4c49eb9c977c6f" member_type: VOTER last_known_addr { host: "127.6.234.193" port: 41331 } } peers { permanent_uuid: "ea80692d30704f638a4646edb9973a82" member_type: VOTER last_known_addr { host: "127.6.234.194" port: 42983 } } peers { permanent_uuid: "55d862e64c3d4bffab828ccb15c5f88d" member_type: VOTER last_known_addr { host: "127.6.234.195" port: 43621 } }
I20260620 11:46:15.674914 25057 leader_election.cc:288] T f194d5eef3fc4d7a97398272db01c8aa P 55d862e64c3d4bffab828ccb15c5f88d [CANDIDATE]: Term 1 election: Requested vote from peers 88dda5569ce2408faa4c49eb9c977c6f (127.6.234.193:41331), ea80692d30704f638a4646edb9973a82 (127.6.234.194:42983)
I20260620 11:46:15.675009 24728 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f194d5eef3fc4d7a97398272db01c8aa" candidate_uuid: "55d862e64c3d4bffab828ccb15c5f88d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "88dda5569ce2408faa4c49eb9c977c6f"
I20260620 11:46:15.675045 24859 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f194d5eef3fc4d7a97398272db01c8aa" candidate_uuid: "55d862e64c3d4bffab828ccb15c5f88d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ea80692d30704f638a4646edb9973a82"
I20260620 11:46:15.675076 24728 raft_consensus.cc:3037] T f194d5eef3fc4d7a97398272db01c8aa P 88dda5569ce2408faa4c49eb9c977c6f [term 0 FOLLOWER]: Advancing to term 1
I20260620 11:46:15.675123 24859 raft_consensus.cc:3037] T f194d5eef3fc4d7a97398272db01c8aa P ea80692d30704f638a4646edb9973a82 [term 0 FOLLOWER]: Advancing to term 1
I20260620 11:46:15.675768 24728 raft_consensus.cc:2454] T f194d5eef3fc4d7a97398272db01c8aa P 88dda5569ce2408faa4c49eb9c977c6f [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 55d862e64c3d4bffab828ccb15c5f88d in term 1.
I20260620 11:46:15.675904 24859 raft_consensus.cc:2454] T f194d5eef3fc4d7a97398272db01c8aa P ea80692d30704f638a4646edb9973a82 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 55d862e64c3d4bffab828ccb15c5f88d in term 1.
I20260620 11:46:15.675959 24923 leader_election.cc:302] T f194d5eef3fc4d7a97398272db01c8aa P 55d862e64c3d4bffab828ccb15c5f88d [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 55d862e64c3d4bffab828ccb15c5f88d, 88dda5569ce2408faa4c49eb9c977c6f; no voters:
I20260620 11:46:15.676055 25057 raft_consensus.cc:2781] T f194d5eef3fc4d7a97398272db01c8aa P 55d862e64c3d4bffab828ccb15c5f88d [term 1 FOLLOWER]: Leader election won for term 1
I20260620 11:46:15.676219 25057 raft_consensus.cc:686] T f194d5eef3fc4d7a97398272db01c8aa P 55d862e64c3d4bffab828ccb15c5f88d [term 1 LEADER]: Becoming Leader. State: Replica: 55d862e64c3d4bffab828ccb15c5f88d, State: Running, Role: LEADER
I20260620 11:46:15.676290 25057 consensus_queue.cc:237] T f194d5eef3fc4d7a97398272db01c8aa P 55d862e64c3d4bffab828ccb15c5f88d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88dda5569ce2408faa4c49eb9c977c6f" member_type: VOTER last_known_addr { host: "127.6.234.193" port: 41331 } } peers { permanent_uuid: "ea80692d30704f638a4646edb9973a82" member_type: VOTER last_known_addr { host: "127.6.234.194" port: 42983 } } peers { permanent_uuid: "55d862e64c3d4bffab828ccb15c5f88d" member_type: VOTER last_known_addr { host: "127.6.234.195" port: 43621 } }
I20260620 11:46:15.677150 24585 catalog_manager.cc:5495] T f194d5eef3fc4d7a97398272db01c8aa P 55d862e64c3d4bffab828ccb15c5f88d reported cstate change: term changed from 0 to 1, leader changed from <none> to 55d862e64c3d4bffab828ccb15c5f88d (127.6.234.195). New cstate: current_term: 1 leader_uuid: "55d862e64c3d4bffab828ccb15c5f88d" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88dda5569ce2408faa4c49eb9c977c6f" member_type: VOTER last_known_addr { host: "127.6.234.193" port: 41331 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "ea80692d30704f638a4646edb9973a82" member_type: VOTER last_known_addr { host: "127.6.234.194" port: 42983 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "55d862e64c3d4bffab828ccb15c5f88d" member_type: VOTER last_known_addr { host: "127.6.234.195" port: 43621 } health_report { overall_health: HEALTHY } } }
W20260620 11:46:15.692003 25037 tablet.cc:2369] T f194d5eef3fc4d7a97398272db01c8aa P 55d862e64c3d4bffab828ccb15c5f88d: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260620 11:46:15.695223 25049 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 11:46:15.692907 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57986 (local address 127.6.234.195:43621)
0620 11:46:15.692950 (+ 43us) server_negotiation.cc:206] Beginning negotiation
0620 11:46:15.692952 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 11:46:15.692961 (+ 9us) server_negotiation.cc:407] Connection header received
0620 11:46:15.693036 (+ 75us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 11:46:15.693036 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 11:46:15.693051 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 11:46:15.693112 (+ 61us) server_negotiation.cc:226] Negotiated authn=TOKEN
0620 11:46:15.693236 (+ 124us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 11:46:15.693793 (+ 557us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 11:46:15.694477 (+ 684us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 11:46:15.694652 (+ 175us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 11:46:15.694701 (+ 49us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 11:46:15.694826 (+ 125us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0620 11:46:15.694915 (+ 89us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0620 11:46:15.694979 (+ 64us) server_negotiation.cc:1035] Waiting for connection context
0620 11:46:15.695096 (+ 117us) server_negotiation.cc:299] Negotiation successful
0620 11:46:15.695112 (+ 16us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":21}
I20260620 11:46:15.697008 24728 raft_consensus.cc:1261] T f194d5eef3fc4d7a97398272db01c8aa P 88dda5569ce2408faa4c49eb9c977c6f [term 1 FOLLOWER]: Refusing update from remote peer 55d862e64c3d4bffab828ccb15c5f88d: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 11:46:15.697008 24859 raft_consensus.cc:1261] T f194d5eef3fc4d7a97398272db01c8aa P ea80692d30704f638a4646edb9973a82 [term 1 FOLLOWER]: Refusing update from remote peer 55d862e64c3d4bffab828ccb15c5f88d: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 11:46:15.697374 25060 consensus_queue.cc:1035] T f194d5eef3fc4d7a97398272db01c8aa P 55d862e64c3d4bffab828ccb15c5f88d [LEADER]: Connected to new peer: Peer: permanent_uuid: "ea80692d30704f638a4646edb9973a82" member_type: VOTER last_known_addr { host: "127.6.234.194" port: 42983 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 11:46:15.697507 25057 consensus_queue.cc:1035] T f194d5eef3fc4d7a97398272db01c8aa P 55d862e64c3d4bffab828ccb15c5f88d [LEADER]: Connected to new peer: Peer: permanent_uuid: "88dda5569ce2408faa4c49eb9c977c6f" member_type: VOTER last_known_addr { host: "127.6.234.193" port: 41331 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 11:46:15.700194 25064 mvcc.cc:205] Tried to move back new op lower bound from 7298891676452200448 to 7298891676370751488. Current Snapshot: MvccSnapshot[applied={T|T < 7298891676452200448}]
I20260620 11:46:15.701902 24585 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:59668:
table { table_name: "test-table" } modify_external_catalogs: true
I20260620 11:46:15.701982 24585 catalog_manager.cc:2645] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:59668:
table { table_name: "test-table" } modify_external_catalogs: true
I20260620 11:46:15.702731 24585 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 4bee377068c6457ba3268e27a513012c: Sending DeleteTablet for 3 replicas of tablet f194d5eef3fc4d7a97398272db01c8aa
I20260620 11:46:15.703068 24839 tablet_service.cc:1514] Processing DeleteTablet for tablet f194d5eef3fc4d7a97398272db01c8aa with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 11:46:15 UTC) from {username='slave'} at 127.0.0.1:40674
I20260620 11:46:15.703055 24970 tablet_service.cc:1514] Processing DeleteTablet for tablet f194d5eef3fc4d7a97398272db01c8aa with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 11:46:15 UTC) from {username='slave'} at 127.0.0.1:57984
I20260620 11:46:15.703279 25074 tablet_replica.cc:330] T f194d5eef3fc4d7a97398272db01c8aa P 55d862e64c3d4bffab828ccb15c5f88d: stopping tablet replica
I20260620 11:46:15.703372 25074 raft_consensus.cc:2229] T f194d5eef3fc4d7a97398272db01c8aa P 55d862e64c3d4bffab828ccb15c5f88d [term 1 LEADER]: Raft consensus shutting down.
I20260620 11:46:15.703467 25074 raft_consensus.cc:2258] T f194d5eef3fc4d7a97398272db01c8aa P 55d862e64c3d4bffab828ccb15c5f88d [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 11:46:15.703531 7083 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task1JRg7N/build/release/bin/kudu with pid 24646
I20260620 11:46:15.704456 25074 ts_tablet_manager.cc:1905] T f194d5eef3fc4d7a97398272db01c8aa P 55d862e64c3d4bffab828ccb15c5f88d: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 11:46:15.705852 25073 tablet_replica.cc:330] T f194d5eef3fc4d7a97398272db01c8aa P ea80692d30704f638a4646edb9973a82: stopping tablet replica
I20260620 11:46:15.705978 25073 raft_consensus.cc:2229] T f194d5eef3fc4d7a97398272db01c8aa P ea80692d30704f638a4646edb9973a82 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260620 11:46:15.705976 25074 ts_tablet_manager.cc:1918] T f194d5eef3fc4d7a97398272db01c8aa P 55d862e64c3d4bffab828ccb15c5f88d: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260620 11:46:15.706034 25074 log.cc:1196] T f194d5eef3fc4d7a97398272db01c8aa P 55d862e64c3d4bffab828ccb15c5f88d: Deleting WAL directory at /tmp/dist-test-task1JRg7N/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781955746769051-7083-0/minicluster-data/ts-2/wal/wals/f194d5eef3fc4d7a97398272db01c8aa
I20260620 11:46:15.706081 25073 raft_consensus.cc:2258] T f194d5eef3fc4d7a97398272db01c8aa P ea80692d30704f638a4646edb9973a82 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 11:46:15.706308 25074 ts_tablet_manager.cc:1939] T f194d5eef3fc4d7a97398272db01c8aa P 55d862e64c3d4bffab828ccb15c5f88d: Deleting consensus metadata
I20260620 11:46:15.706478 25073 ts_tablet_manager.cc:1905] T f194d5eef3fc4d7a97398272db01c8aa P ea80692d30704f638a4646edb9973a82: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 11:46:15.706677 24570 catalog_manager.cc:4841] TS 55d862e64c3d4bffab828ccb15c5f88d (127.6.234.195:43621): tablet f194d5eef3fc4d7a97398272db01c8aa (table test-table [id=a29909ce88314fe4af2658f4135ce565]) successfully deleted
I20260620 11:46:15.707726 25073 ts_tablet_manager.cc:1918] T f194d5eef3fc4d7a97398272db01c8aa P ea80692d30704f638a4646edb9973a82: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260620 11:46:15.707774 25073 log.cc:1196] T f194d5eef3fc4d7a97398272db01c8aa P ea80692d30704f638a4646edb9973a82: Deleting WAL directory at /tmp/dist-test-task1JRg7N/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781955746769051-7083-0/minicluster-data/ts-1/wal/wals/f194d5eef3fc4d7a97398272db01c8aa
I20260620 11:46:15.707963 25073 ts_tablet_manager.cc:1939] T f194d5eef3fc4d7a97398272db01c8aa P ea80692d30704f638a4646edb9973a82: Deleting consensus metadata
I20260620 11:46:15.708279 24573 catalog_manager.cc:4841] TS ea80692d30704f638a4646edb9973a82 (127.6.234.194:42983): tablet f194d5eef3fc4d7a97398272db01c8aa (table test-table [id=a29909ce88314fe4af2658f4135ce565]) successfully deleted
W20260620 11:46:15.710656 24570 connection.cc:535] client connection to 127.6.234.193:41331 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
W20260620 11:46:15.710772 24570 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
I20260620 11:46:15.710927 7083 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task1JRg7N/build/release/bin/kudu with pid 24777
I20260620 11:46:15.711711 25045 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 11:46:15.711577 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.234.193:41331 (local address 127.0.0.1:55756)
0620 11:46:15.711636 (+ 59us) negotiation.cc:107] Waiting for socket to connect
0620 11:46:15.711679 (+ 43us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.234.193:41331: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":37}
W20260620 11:46:15.711897 24570 catalog_manager.cc:4584] TS 88dda5569ce2408faa4c49eb9c977c6f (127.6.234.193:41331): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet f194d5eef3fc4d7a97398272db01c8aa: Network error: Client connection negotiation failed: client connection to 127.6.234.193:41331: connect: Connection refused (error 111)
I20260620 11:46:15.717038 7083 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task1JRg7N/build/release/bin/kudu with pid 24908
I20260620 11:46:15.722142 7083 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task1JRg7N/build/release/bin/kudu with pid 24555
2026-06-20T11:46:15Z chronyd exiting
[ OK ] EncryptionPolicyTest.TestEncryptionPolicy/1 (853 ms)
[ RUN ] EncryptionPolicyTest.TestEncryptionPolicy/2
Loading random data
Initializing database '/tmp/dist-test-task1JRg7N/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781955746769051-7083-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 20 11:46:15 dist-test-slave-cjc6 krb5kdc[25078](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 20 11:46:15 dist-test-slave-cjc6 krb5kdc[25078](info): set up 2 sockets
Jun 20 11:46:15 dist-test-slave-cjc6 krb5kdc[25078](info): commencing operation
krb5kdc: starting...
W20260620 11:46:16.749491 7083 mini_kdc.cc:121] Time spent starting KDC: real 1.008s user 0.000s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task1JRg7N/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781955746769051-7083-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task1JRg7N/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781955746769051-7083-0/krb5kdc/test-user.keytab.
Jun 20 11:46:16 dist-test-slave-cjc6 krb5kdc[25078](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781955976, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-20T11:46:16Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-20T11:46:16Z Disabled control of system clock
WARNING: no policy specified for kudu/127.6.234.254@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.6.234.254@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.6.234.254 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task1JRg7N/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781955746769051-7083-0/krb5kdc/kudu_127.6.234.254.keytab.
Entry for principal kudu/127.6.234.254 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task1JRg7N/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781955746769051-7083-0/krb5kdc/kudu_127.6.234.254.keytab.
I20260620 11:46:16.864825 7083 external_mini_cluster.cc:1326] Running /tmp/dist-test-task1JRg7N/build/release/bin/kudu
/tmp/dist-test-task1JRg7N/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task1JRg7N/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781955746769051-7083-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task1JRg7N/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781955746769051-7083-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task1JRg7N/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781955746769051-7083-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task1JRg7N/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781955746769051-7083-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.6.234.254:35111
--webserver_interface=127.6.234.254
--webserver_port=0
--keytab_file=/tmp/dist-test-task1JRg7N/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781955746769051-7083-0/krb5kdc/kudu_127.6.234.254.keytab
--principal=kudu/127.6.234.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.6.234.212:36351
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.6.234.254:35111
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task1JRg7N/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781955746769051-7083-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task1JRg7N/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781955746769051-7083-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task1JRg7N/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781955746769051-7083-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 11:46:16.944336 25091 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 11:46:16.944592 25091 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 11:46:16.944630 25091 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 11:46:16.945849 25091 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 11:46:16.945924 25091 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 11:46:16.945950 25091 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 11:46:16.945972 25091 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 11:46:16.945999 25091 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 11:46:16.947469 25091 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.6.234.212:36351
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task1JRg7N/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781955746769051-7083-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task1JRg7N/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781955746769051-7083-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.6.234.254:35111
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task1JRg7N/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781955746769051-7083-0/krb5kdc/kudu_127.6.234.254.keytab
--principal=kudu/127.6.234.254
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.6.234.254:35111
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task1JRg7N/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781955746769051-7083-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.6.234.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.25091
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task1JRg7N/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781955746769051-7083-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 20 Jun 2026 10:32:37 UTC on ccycloud.quasar-bgyofv.root.comops.site
build id 4457173
I20260620 11:46:16.947755 25091 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 11:46:16.947984 25091 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260620 11:46:16.952831 25091 server_base.cc:1034] running on GCE node
W20260620 11:46:16.952782 25097 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 11:46:16.952828 25096 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 11:46:16.952782 25099 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 11:46:16.953346 25091 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 11:46:16.953639 25091 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 11:46:16.954807 25091 hybrid_clock.cc:648] HybridClock initialized: now 1781955976954787 us; error 38 us; skew 500 ppm
Jun 20 11:46:16 dist-test-slave-cjc6 krb5kdc[25078](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781955976, etypes {rep=17 tkt=17 ses=17}, kudu/127.6.234.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 11:46:16.957234 25091 init.cc:377] Logged in from keytab as kudu/127.6.234.254@KRBTEST.COM (short username kudu)
I20260620 11:46:16.957734 25091 webserver.cc:460] Webserver started at http://127.6.234.254:43479/ using document root <none> and password file <none>
I20260620 11:46:16.958048 25091 fs_manager.cc:359] Metadata directory not provided
I20260620 11:46:16.958139 25091 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 11:46:16.958292 25091 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 11:46:16.959386 25091 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task1JRg7N/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781955746769051-7083-0/minicluster-data/master-0/data/instance:
uuid: "0d852506e1ef4719b025ac0a447a18dd"
format_stamp: "Formatted at 2026-06-20 11:46:16 on dist-test-slave-cjc6"
server_key: "cce63daf8ce188694dfaecfef538aad0"
server_key_iv: "6edaa55116516f80208511b127129dc8"
server_key_version: "encryptionkey@0"
I20260620 11:46:16.959684 25091 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task1JRg7N/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781955746769051-7083-0/minicluster-data/master-0/wal/instance:
uuid: "0d852506e1ef4719b025ac0a447a18dd"
format_stamp: "Formatted at 2026-06-20 11:46:16 on dist-test-slave-cjc6"
server_key: "cce63daf8ce188694dfaecfef538aad0"
server_key_iv: "6edaa55116516f80208511b127129dc8"
server_key_version: "encryptionkey@0"
I20260620 11:46:16.960965 25091 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260620 11:46:16.961665 25106 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 11:46:16.961903 25091 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260620 11:46:16.962011 25091 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task1JRg7N/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781955746769051-7083-0/minicluster-data/master-0/data,/tmp/dist-test-task1JRg7N/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781955746769051-7083-0/minicluster-data/master-0/wal
uuid: "0d852506e1ef4719b025ac0a447a18dd"
format_stamp: "Formatted at 2026-06-20 11:46:16 on dist-test-slave-cjc6"
server_key: "cce63daf8ce188694dfaecfef538aad0"
server_key_iv: "6edaa55116516f80208511b127129dc8"
server_key_version: "encryptionkey@0"
I20260620 11:46:16.962103 25091 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task1JRg7N/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781955746769051-7083-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task1JRg7N/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781955746769051-7083-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task1JRg7N/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781955746769051-7083-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 11:46:16.970654 25091 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 11:46:16.972380 25091 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 11:46:16.972549 25091 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 11:46:16.976377 25091 rpc_server.cc:307] RPC server started. Bound to: 127.6.234.254:35111
I20260620 11:46:16.976428 25158 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.6.234.254:35111 every 8 connection(s)
I20260620 11:46:16.976822 25091 server_base.cc:1166] Dumped server information to /tmp/dist-test-task1JRg7N/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781955746769051-7083-0/minicluster-data/master-0/data/info.pb
I20260620 11:46:16.977685 25159 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 11:46:16.979451 7083 external_mini_cluster.cc:1388] Started /tmp/dist-test-task1JRg7N/build/release/bin/kudu as pid 25091
I20260620 11:46:16.979604 7083 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task1JRg7N/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781955746769051-7083-0/minicluster-data/master-0/wal/instance
I20260620 11:46:16.979804 7083 external_mini_cluster.cc:1428] Setting key e6cc1785a6cba24367d0c6d4df1280fa
I20260620 11:46:16.980456 25159 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 0d852506e1ef4719b025ac0a447a18dd: Bootstrap starting.
I20260620 11:46:16.981060 25159 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 0d852506e1ef4719b025ac0a447a18dd: Neither blocks nor log segments found. Creating new log.
I20260620 11:46:16.981390 25159 log.cc:824] T 00000000000000000000000000000000 P 0d852506e1ef4719b025ac0a447a18dd: Log is configured to *not* fsync() on all Append() calls
I20260620 11:46:16.982184 25159 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 0d852506e1ef4719b025ac0a447a18dd: No bootstrap required, opened a new log
I20260620 11:46:16.984292 25159 raft_consensus.cc:348] T 00000000000000000000000000000000 P 0d852506e1ef4719b025ac0a447a18dd [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0d852506e1ef4719b025ac0a447a18dd" member_type: VOTER last_known_addr { host: "127.6.234.254" port: 35111 } }
I20260620 11:46:16.984422 25159 raft_consensus.cc:374] T 00000000000000000000000000000000 P 0d852506e1ef4719b025ac0a447a18dd [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 11:46:16.984444 25159 raft_consensus.cc:729] T 00000000000000000000000000000000 P 0d852506e1ef4719b025ac0a447a18dd [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0d852506e1ef4719b025ac0a447a18dd, State: Initialized, Role: FOLLOWER
I20260620 11:46:16.984579 25159 consensus_queue.cc:260] T 00000000000000000000000000000000 P 0d852506e1ef4719b025ac0a447a18dd [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0d852506e1ef4719b025ac0a447a18dd" member_type: VOTER last_known_addr { host: "127.6.234.254" port: 35111 } }
I20260620 11:46:16.984678 25159 raft_consensus.cc:388] T 00000000000000000000000000000000 P 0d852506e1ef4719b025ac0a447a18dd [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260620 11:46:16.984710 25159 raft_consensus.cc:482] T 00000000000000000000000000000000 P 0d852506e1ef4719b025ac0a447a18dd [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260620 11:46:16.984797 25159 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 0d852506e1ef4719b025ac0a447a18dd [term 0 FOLLOWER]: Advancing to term 1
I20260620 11:46:16.985476 25159 raft_consensus.cc:504] T 00000000000000000000000000000000 P 0d852506e1ef4719b025ac0a447a18dd [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0d852506e1ef4719b025ac0a447a18dd" member_type: VOTER last_known_addr { host: "127.6.234.254" port: 35111 } }
I20260620 11:46:16.985594 25159 leader_election.cc:302] T 00000000000000000000000000000000 P 0d852506e1ef4719b025ac0a447a18dd [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 0d852506e1ef4719b025ac0a447a18dd; no voters:
I20260620 11:46:16.985790 25159 leader_election.cc:288] T 00000000000000000000000000000000 P 0d852506e1ef4719b025ac0a447a18dd [CANDIDATE]: Term 1 election: Requested vote from peers
I20260620 11:46:16.985865 25164 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 0d852506e1ef4719b025ac0a447a18dd [term 1 FOLLOWER]: Leader election won for term 1
I20260620 11:46:16.986055 25159 sys_catalog.cc:564] T 00000000000000000000000000000000 P 0d852506e1ef4719b025ac0a447a18dd [sys.catalog]: configured and running, proceeding with master startup.
I20260620 11:46:16.986057 25164 raft_consensus.cc:686] T 00000000000000000000000000000000 P 0d852506e1ef4719b025ac0a447a18dd [term 1 LEADER]: Becoming Leader. State: Replica: 0d852506e1ef4719b025ac0a447a18dd, State: Running, Role: LEADER
I20260620 11:46:16.986245 25164 consensus_queue.cc:237] T 00000000000000000000000000000000 P 0d852506e1ef4719b025ac0a447a18dd [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0d852506e1ef4719b025ac0a447a18dd" member_type: VOTER last_known_addr { host: "127.6.234.254" port: 35111 } }
I20260620 11:46:16.986644 25166 sys_catalog.cc:455] T 00000000000000000000000000000000 P 0d852506e1ef4719b025ac0a447a18dd [sys.catalog]: SysCatalogTable state changed. Reason: New leader 0d852506e1ef4719b025ac0a447a18dd. Latest consensus state: current_term: 1 leader_uuid: "0d852506e1ef4719b025ac0a447a18dd" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0d852506e1ef4719b025ac0a447a18dd" member_type: VOTER last_known_addr { host: "127.6.234.254" port: 35111 } } }
I20260620 11:46:16.986726 25166 sys_catalog.cc:458] T 00000000000000000000000000000000 P 0d852506e1ef4719b025ac0a447a18dd [sys.catalog]: This master's current role is: LEADER
Jun 20 11:46:16 dist-test-slave-cjc6 krb5kdc[25078](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781955976, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.6.234.254@KRBTEST.COM
I20260620 11:46:16.986950 25165 sys_catalog.cc:455] T 00000000000000000000000000000000 P 0d852506e1ef4719b025ac0a447a18dd [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "0d852506e1ef4719b025ac0a447a18dd" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0d852506e1ef4719b025ac0a447a18dd" member_type: VOTER last_known_addr { host: "127.6.234.254" port: 35111 } } }
I20260620 11:46:16.987291 25165 sys_catalog.cc:458] T 00000000000000000000000000000000 P 0d852506e1ef4719b025ac0a447a18dd [sys.catalog]: This master's current role is: LEADER
I20260620 11:46:16.987432 25173 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260620 11:46:16.988468 25173 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260620 11:46:16.990233 25173 catalog_manager.cc:1347] Generated new cluster ID: 59a8f7059d9740d49a7092b1c0c0c1a0
I20260620 11:46:16.990288 25173 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260620 11:46:16.991084 25162 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 11:46:16.980312 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56924 (local address 127.6.234.254:35111)
0620 11:46:16.980495 (+ 183us) server_negotiation.cc:206] Beginning negotiation
0620 11:46:16.980503 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0620 11:46:16.980552 (+ 49us) server_negotiation.cc:407] Connection header received
0620 11:46:16.980986 (+ 434us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 11:46:16.980993 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 11:46:16.981038 (+ 45us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 11:46:16.981199 (+ 161us) server_negotiation.cc:226] Negotiated authn=SASL
0620 11:46:16.982030 (+ 831us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 11:46:16.982954 (+ 924us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 11:46:16.984103 (+ 1149us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 11:46:16.984588 (+ 485us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 11:46:16.984704 (+ 116us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 11:46:16.987626 (+ 2922us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 11:46:16.987630 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 11:46:16.987631 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 11:46:16.987642 (+ 11us) server_negotiation.cc:922] calling sasl_server_start()
0620 11:46:16.989435 (+ 1793us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 11:46:16.989854 (+ 419us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 11:46:16.989856 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 11:46:16.989857 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 11:46:16.989938 (+ 81us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 11:46:16.990196 (+ 258us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 11:46:16.990198 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 11:46:16.990198 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 11:46:16.990496 (+ 298us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 11:46:16.990657 (+ 161us) server_negotiation.cc:1035] Waiting for connection context
0620 11:46:16.990763 (+ 106us) server_negotiation.cc:299] Negotiation successful
0620 11:46:16.990900 (+ 137us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":124,"thread_start_us":92,"threads_started":1}
I20260620 11:46:17.000985 25173 catalog_manager.cc:1370] Generated new certificate authority record
I20260620 11:46:17.001654 25173 catalog_manager.cc:1504] Loading token signing keys...
I20260620 11:46:17.005026 25173 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 0d852506e1ef4719b025ac0a447a18dd: Generated new TSK 0
I20260620 11:46:17.005206 25173 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.6.234.193@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.6.234.193@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.6.234.193 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task1JRg7N/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781955746769051-7083-0/krb5kdc/kudu_127.6.234.193.keytab.
Entry for principal kudu/127.6.234.193 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task1JRg7N/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781955746769051-7083-0/krb5kdc/kudu_127.6.234.193.keytab.
I20260620 11:46:17.037027 7083 external_mini_cluster.cc:1326] Running /tmp/dist-test-task1JRg7N/build/release/bin/kudu
/tmp/dist-test-task1JRg7N/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task1JRg7N/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781955746769051-7083-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-task1JRg7N/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781955746769051-7083-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task1JRg7N/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781955746769051-7083-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-task1JRg7N/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781955746769051-7083-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.6.234.193:0
--local_ip_for_outbound_sockets=127.6.234.193
--webserver_interface=127.6.234.193
--webserver_port=0
--tserver_master_addrs=127.6.234.254:35111
--keytab_file=/tmp/dist-test-task1JRg7N/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781955746769051-7083-0/krb5kdc/kudu_127.6.234.193.keytab
--principal=kudu/127.6.234.193
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.6.234.212:36351
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task1JRg7N/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781955746769051-7083-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task1JRg7N/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781955746769051-7083-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task1JRg7N/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781955746769051-7083-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 11:46:17.154038 25185 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 11:46:17.154309 25185 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 11:46:17.154341 25185 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 11:46:17.156020 25185 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 11:46:17.156080 25185 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 11:46:17.156119 25185 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.6.234.193
I20260620 11:46:17.158342 25185 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.6.234.212:36351
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task1JRg7N/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781955746769051-7083-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-task1JRg7N/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781955746769051-7083-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task1JRg7N/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781955746769051-7083-0/krb5kdc/kudu_127.6.234.193.keytab
--principal=kudu/127.6.234.193
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.6.234.193:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task1JRg7N/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781955746769051-7083-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.6.234.193
--webserver_port=0
--tserver_master_addrs=127.6.234.254:35111
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.25185
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.6.234.193
--log_dir=/tmp/dist-test-task1JRg7N/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781955746769051-7083-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 20 Jun 2026 10:32:37 UTC on ccycloud.quasar-bgyofv.root.comops.site
build id 4457173
I20260620 11:46:17.158586 25185 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 11:46:17.158838 25185 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 11:46:17.164126 25190 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 11:46:17.164142 25193 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 11:46:17.164222 25191 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 11:46:17.164667 25185 server_base.cc:1034] running on GCE node
I20260620 11:46:17.164901 25185 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 11:46:17.165146 25185 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 11:46:17.166312 25185 hybrid_clock.cc:648] HybridClock initialized: now 1781955977166283 us; error 44 us; skew 500 ppm
Jun 20 11:46:17 dist-test-slave-cjc6 krb5kdc[25078](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781955977, etypes {rep=17 tkt=17 ses=17}, kudu/127.6.234.193@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 11:46:17.168923 25185 init.cc:377] Logged in from keytab as kudu/127.6.234.193@KRBTEST.COM (short username kudu)
I20260620 11:46:17.169410 25185 webserver.cc:460] Webserver started at http://127.6.234.193:44291/ using document root <none> and password file <none>
I20260620 11:46:17.169664 25185 fs_manager.cc:359] Metadata directory not provided
I20260620 11:46:17.169757 25185 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 11:46:17.169903 25185 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 11:46:17.170946 25185 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task1JRg7N/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781955746769051-7083-0/minicluster-data/ts-0/data/instance:
uuid: "65d7c74f35e648e181d1eb2f8419924e"
format_stamp: "Formatted at 2026-06-20 11:46:17 on dist-test-slave-cjc6"
server_key: "d997e70e00b31e464975e4239788fc4a"
server_key_iv: "d53a41132bb14c874ee136f6d8102eed"
server_key_version: "encryptionkey@0"
I20260620 11:46:17.171308 25185 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task1JRg7N/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781955746769051-7083-0/minicluster-data/ts-0/wal/instance:
uuid: "65d7c74f35e648e181d1eb2f8419924e"
format_stamp: "Formatted at 2026-06-20 11:46:17 on dist-test-slave-cjc6"
server_key: "d997e70e00b31e464975e4239788fc4a"
server_key_iv: "d53a41132bb14c874ee136f6d8102eed"
server_key_version: "encryptionkey@0"
I20260620 11:46:17.172631 25185 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.003s
I20260620 11:46:17.173360 25200 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 11:46:17.173508 25185 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.001s
I20260620 11:46:17.173615 25185 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task1JRg7N/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781955746769051-7083-0/minicluster-data/ts-0/data,/tmp/dist-test-task1JRg7N/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781955746769051-7083-0/minicluster-data/ts-0/wal
uuid: "65d7c74f35e648e181d1eb2f8419924e"
format_stamp: "Formatted at 2026-06-20 11:46:17 on dist-test-slave-cjc6"
server_key: "d997e70e00b31e464975e4239788fc4a"
server_key_iv: "d53a41132bb14c874ee136f6d8102eed"
server_key_version: "encryptionkey@0"
I20260620 11:46:17.173691 25185 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task1JRg7N/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781955746769051-7083-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-task1JRg7N/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781955746769051-7083-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-task1JRg7N/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781955746769051-7083-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 11:46:17.187923 25185 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 11:46:17.190011 25185 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 11:46:17.190188 25185 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 11:46:17.190433 25185 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 11:46:17.190744 25185 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 11:46:17.190800 25185 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 11:46:17.190846 25185 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 11:46:17.190879 25185 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 11:46:17.196796 25185 rpc_server.cc:307] RPC server started. Bound to: 127.6.234.193:38993
I20260620 11:46:17.196869 25313 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.6.234.193:38993 every 8 connection(s)
I20260620 11:46:17.197196 25185 server_base.cc:1166] Dumped server information to /tmp/dist-test-task1JRg7N/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781955746769051-7083-0/minicluster-data/ts-0/data/info.pb
I20260620 11:46:17.202843 7083 external_mini_cluster.cc:1388] Started /tmp/dist-test-task1JRg7N/build/release/bin/kudu as pid 25185
I20260620 11:46:17.202976 7083 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task1JRg7N/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781955746769051-7083-0/minicluster-data/ts-0/wal/instance
I20260620 11:46:17.203166 7083 external_mini_cluster.cc:1428] Setting key f3bdcd242a99346c635fce09bda2d660
Jun 20 11:46:17 dist-test-slave-cjc6 krb5kdc[25078](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781955977, etypes {rep=17 tkt=17 ses=17}, kudu/127.6.234.193@KRBTEST.COM for kudu/127.6.234.254@KRBTEST.COM
I20260620 11:46:17.207237 25162 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 11:46:17.197919 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.6.234.193:56121 (local address 127.6.234.254:35111)
0620 11:46:17.197967 (+ 48us) server_negotiation.cc:206] Beginning negotiation
0620 11:46:17.197969 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 11:46:17.198178 (+ 209us) server_negotiation.cc:407] Connection header received
0620 11:46:17.198729 (+ 551us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 11:46:17.198731 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 11:46:17.198744 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 11:46:17.198796 (+ 52us) server_negotiation.cc:226] Negotiated authn=SASL
0620 11:46:17.199758 (+ 962us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 11:46:17.200279 (+ 521us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 11:46:17.201344 (+ 1065us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 11:46:17.201561 (+ 217us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 11:46:17.201650 (+ 89us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 11:46:17.204025 (+ 2375us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 11:46:17.204026 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 11:46:17.204027 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 11:46:17.204034 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0620 11:46:17.205578 (+ 1544us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 11:46:17.206189 (+ 611us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 11:46:17.206190 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 11:46:17.206191 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 11:46:17.206249 (+ 58us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 11:46:17.206537 (+ 288us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 11:46:17.206538 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 11:46:17.206538 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 11:46:17.206641 (+ 103us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 11:46:17.206692 (+ 51us) server_negotiation.cc:1035] Waiting for connection context
0620 11:46:17.207079 (+ 387us) server_negotiation.cc:299] Negotiation successful
0620 11:46:17.207168 (+ 89us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":20}
I20260620 11:46:17.207509 25316 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 11:46:17.197867 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.234.254:35111 (local address 127.6.234.193:56121)
0620 11:46:17.198043 (+ 176us) negotiation.cc:107] Waiting for socket to connect
0620 11:46:17.198079 (+ 36us) client_negotiation.cc:174] Beginning negotiation
0620 11:46:17.198609 (+ 530us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 11:46:17.198868 (+ 259us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 11:46:17.198872 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 11:46:17.199448 (+ 576us) client_negotiation.cc:189] Negotiated authn=SASL
0620 11:46:17.199650 (+ 202us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 11:46:17.199654 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 11:46:17.200381 (+ 727us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 11:46:17.200382 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 11:46:17.201256 (+ 874us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 11:46:17.201259 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 11:46:17.201661 (+ 402us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 11:46:17.201662 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 11:46:17.201830 (+ 168us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 11:46:17.202315 (+ 485us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 11:46:17.202322 (+ 7us) client_negotiation.cc:656] calling sasl_client_start()
0620 11:46:17.203923 (+ 1601us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 11:46:17.205666 (+ 1743us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 11:46:17.205673 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 11:46:17.205675 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0620 11:46:17.206111 (+ 436us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 11:46:17.206323 (+ 212us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 11:46:17.206324 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 11:46:17.206324 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0620 11:46:17.206472 (+ 148us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 11:46:17.206707 (+ 235us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 11:46:17.206713 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 11:46:17.206888 (+ 175us) client_negotiation.cc:769] Sending connection context
0620 11:46:17.207018 (+ 130us) client_negotiation.cc:240] Negotiation successful
0620 11:46:17.207215 (+ 197us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":138,"thread_start_us":71,"threads_started":1}
I20260620 11:46:17.208101 25314 heartbeater.cc:347] Connected to a master server at 127.6.234.254:35111
I20260620 11:46:17.208213 25314 heartbeater.cc:464] Registering TS with master...
I20260620 11:46:17.208422 25314 heartbeater.cc:511] Master 127.6.234.254:35111 requested a full tablet report, sending...
I20260620 11:46:17.208972 25123 ts_manager.cc:194] Registered new tserver with Master: 65d7c74f35e648e181d1eb2f8419924e (127.6.234.193:38993)
I20260620 11:46:17.209605 25123 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.6.234.193@KRBTEST.COM'} at 127.6.234.193:56121
WARNING: no policy specified for kudu/127.6.234.194@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.6.234.194@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.6.234.194 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task1JRg7N/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781955746769051-7083-0/krb5kdc/kudu_127.6.234.194.keytab.
Entry for principal kudu/127.6.234.194 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task1JRg7N/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781955746769051-7083-0/krb5kdc/kudu_127.6.234.194.keytab.
I20260620 11:46:17.228271 7083 external_mini_cluster.cc:1326] Running /tmp/dist-test-task1JRg7N/build/release/bin/kudu
/tmp/dist-test-task1JRg7N/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task1JRg7N/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781955746769051-7083-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-task1JRg7N/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781955746769051-7083-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task1JRg7N/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781955746769051-7083-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-task1JRg7N/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781955746769051-7083-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.6.234.194:0
--local_ip_for_outbound_sockets=127.6.234.194
--webserver_interface=127.6.234.194
--webserver_port=0
--tserver_master_addrs=127.6.234.254:35111
--keytab_file=/tmp/dist-test-task1JRg7N/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781955746769051-7083-0/krb5kdc/kudu_127.6.234.194.keytab
--principal=kudu/127.6.234.194
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.6.234.212:36351
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task1JRg7N/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781955746769051-7083-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task1JRg7N/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781955746769051-7083-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task1JRg7N/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781955746769051-7083-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 11:46:17.309178 25319 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 11:46:17.309377 25319 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 11:46:17.309401 25319 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 11:46:17.310658 25319 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 11:46:17.310710 25319 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 11:46:17.310741 25319 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.6.234.194
I20260620 11:46:17.312260 25319 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.6.234.212:36351
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task1JRg7N/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781955746769051-7083-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-task1JRg7N/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781955746769051-7083-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task1JRg7N/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781955746769051-7083-0/krb5kdc/kudu_127.6.234.194.keytab
--principal=kudu/127.6.234.194
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.6.234.194:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task1JRg7N/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781955746769051-7083-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.6.234.194
--webserver_port=0
--tserver_master_addrs=127.6.234.254:35111
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.25319
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.6.234.194
--log_dir=/tmp/dist-test-task1JRg7N/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781955746769051-7083-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 20 Jun 2026 10:32:37 UTC on ccycloud.quasar-bgyofv.root.comops.site
build id 4457173
I20260620 11:46:17.312511 25319 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 11:46:17.312757 25319 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 11:46:17.317741 25324 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 11:46:17.317751 25325 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 11:46:17.317852 25319 server_base.cc:1034] running on GCE node
W20260620 11:46:17.317772 25327 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 11:46:17.318226 25319 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 11:46:17.318470 25319 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 11:46:17.319636 25319 hybrid_clock.cc:648] HybridClock initialized: now 1781955977319613 us; error 33 us; skew 500 ppm
Jun 20 11:46:17 dist-test-slave-cjc6 krb5kdc[25078](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781955977, etypes {rep=17 tkt=17 ses=17}, kudu/127.6.234.194@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 11:46:17.322114 25319 init.cc:377] Logged in from keytab as kudu/127.6.234.194@KRBTEST.COM (short username kudu)
I20260620 11:46:17.322576 25319 webserver.cc:460] Webserver started at http://127.6.234.194:32999/ using document root <none> and password file <none>
I20260620 11:46:17.322814 25319 fs_manager.cc:359] Metadata directory not provided
I20260620 11:46:17.322883 25319 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 11:46:17.323035 25319 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 11:46:17.324036 25319 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task1JRg7N/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781955746769051-7083-0/minicluster-data/ts-1/data/instance:
uuid: "789253646a8442f985c63e3d47306976"
format_stamp: "Formatted at 2026-06-20 11:46:17 on dist-test-slave-cjc6"
server_key: "48a86d77a66991b584c8347bd30a255b"
server_key_iv: "c62f44e2ccf96efa4dd32367806806be"
server_key_version: "encryptionkey@0"
I20260620 11:46:17.324388 25319 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task1JRg7N/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781955746769051-7083-0/minicluster-data/ts-1/wal/instance:
uuid: "789253646a8442f985c63e3d47306976"
format_stamp: "Formatted at 2026-06-20 11:46:17 on dist-test-slave-cjc6"
server_key: "48a86d77a66991b584c8347bd30a255b"
server_key_iv: "c62f44e2ccf96efa4dd32367806806be"
server_key_version: "encryptionkey@0"
I20260620 11:46:17.325630 25319 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260620 11:46:17.326378 25334 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 11:46:17.326560 25319 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260620 11:46:17.326655 25319 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task1JRg7N/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781955746769051-7083-0/minicluster-data/ts-1/data,/tmp/dist-test-task1JRg7N/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781955746769051-7083-0/minicluster-data/ts-1/wal
uuid: "789253646a8442f985c63e3d47306976"
format_stamp: "Formatted at 2026-06-20 11:46:17 on dist-test-slave-cjc6"
server_key: "48a86d77a66991b584c8347bd30a255b"
server_key_iv: "c62f44e2ccf96efa4dd32367806806be"
server_key_version: "encryptionkey@0"
I20260620 11:46:17.326740 25319 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task1JRg7N/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781955746769051-7083-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-task1JRg7N/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781955746769051-7083-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-task1JRg7N/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781955746769051-7083-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 11:46:17.354763 25319 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 11:46:17.356619 25319 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 11:46:17.356794 25319 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 11:46:17.357056 25319 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 11:46:17.357371 25319 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 11:46:17.357429 25319 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 11:46:17.357470 25319 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 11:46:17.357492 25319 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 11:46:17.363296 25319 rpc_server.cc:307] RPC server started. Bound to: 127.6.234.194:46385
I20260620 11:46:17.363351 25447 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.6.234.194:46385 every 8 connection(s)
I20260620 11:46:17.363646 25319 server_base.cc:1166] Dumped server information to /tmp/dist-test-task1JRg7N/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781955746769051-7083-0/minicluster-data/ts-1/data/info.pb
Jun 20 11:46:17 dist-test-slave-cjc6 krb5kdc[25078](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781955977, etypes {rep=17 tkt=17 ses=17}, kudu/127.6.234.194@KRBTEST.COM for kudu/127.6.234.254@KRBTEST.COM
I20260620 11:46:17.372752 25162 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 11:46:17.364287 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.6.234.194:50055 (local address 127.6.234.254:35111)
0620 11:46:17.364330 (+ 43us) server_negotiation.cc:206] Beginning negotiation
0620 11:46:17.364332 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 11:46:17.364639 (+ 307us) server_negotiation.cc:407] Connection header received
0620 11:46:17.365197 (+ 558us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 11:46:17.365198 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 11:46:17.365211 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 11:46:17.365252 (+ 41us) server_negotiation.cc:226] Negotiated authn=SASL
0620 11:46:17.366271 (+ 1019us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 11:46:17.366786 (+ 515us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 11:46:17.367781 (+ 995us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 11:46:17.367989 (+ 208us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 11:46:17.368069 (+ 80us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 11:46:17.370177 (+ 2108us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 11:46:17.370178 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 11:46:17.370179 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 11:46:17.370185 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0620 11:46:17.371471 (+ 1286us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 11:46:17.371878 (+ 407us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 11:46:17.371879 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 11:46:17.371879 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 11:46:17.371930 (+ 51us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 11:46:17.372203 (+ 273us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 11:46:17.372204 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 11:46:17.372204 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 11:46:17.372320 (+ 116us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 11:46:17.372376 (+ 56us) server_negotiation.cc:1035] Waiting for connection context
0620 11:46:17.372620 (+ 244us) server_negotiation.cc:299] Negotiation successful
0620 11:46:17.372688 (+ 68us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":19}
I20260620 11:46:17.372938 25450 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 11:46:17.364307 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.234.254:35111 (local address 127.6.234.194:50055)
0620 11:46:17.364491 (+ 184us) negotiation.cc:107] Waiting for socket to connect
0620 11:46:17.364559 (+ 68us) client_negotiation.cc:174] Beginning negotiation
0620 11:46:17.365061 (+ 502us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 11:46:17.365333 (+ 272us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 11:46:17.365337 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 11:46:17.365980 (+ 643us) client_negotiation.cc:189] Negotiated authn=SASL
0620 11:46:17.366173 (+ 193us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 11:46:17.366178 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 11:46:17.366886 (+ 708us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 11:46:17.366887 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 11:46:17.367688 (+ 801us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 11:46:17.367690 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 11:46:17.368076 (+ 386us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 11:46:17.368077 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 11:46:17.368209 (+ 132us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 11:46:17.368669 (+ 460us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 11:46:17.368677 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0620 11:46:17.370091 (+ 1414us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 11:46:17.371565 (+ 1474us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 11:46:17.371571 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 11:46:17.371573 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0620 11:46:17.371793 (+ 220us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 11:46:17.372017 (+ 224us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 11:46:17.372018 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 11:46:17.372018 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0620 11:46:17.372130 (+ 112us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 11:46:17.372384 (+ 254us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 11:46:17.372388 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 11:46:17.372509 (+ 121us) client_negotiation.cc:769] Sending connection context
0620 11:46:17.372593 (+ 84us) client_negotiation.cc:240] Negotiation successful
0620 11:46:17.372715 (+ 122us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":141,"thread_start_us":91,"threads_started":1}
I20260620 11:46:17.373147 7083 external_mini_cluster.cc:1388] Started /tmp/dist-test-task1JRg7N/build/release/bin/kudu as pid 25319
I20260620 11:46:17.373297 7083 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task1JRg7N/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781955746769051-7083-0/minicluster-data/ts-1/wal/instance
I20260620 11:46:17.373435 25448 heartbeater.cc:347] Connected to a master server at 127.6.234.254:35111
I20260620 11:46:17.373529 7083 external_mini_cluster.cc:1428] Setting key 6282475d8c43bb9faee21e51f9200f71
I20260620 11:46:17.373546 25448 heartbeater.cc:464] Registering TS with master...
I20260620 11:46:17.373761 25448 heartbeater.cc:511] Master 127.6.234.254:35111 requested a full tablet report, sending...
I20260620 11:46:17.374225 25123 ts_manager.cc:194] Registered new tserver with Master: 789253646a8442f985c63e3d47306976 (127.6.234.194:46385)
I20260620 11:46:17.374594 25123 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.6.234.194@KRBTEST.COM'} at 127.6.234.194:50055
WARNING: no policy specified for kudu/127.6.234.195@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.6.234.195@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.6.234.195 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task1JRg7N/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781955746769051-7083-0/krb5kdc/kudu_127.6.234.195.keytab.
Entry for principal kudu/127.6.234.195 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task1JRg7N/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781955746769051-7083-0/krb5kdc/kudu_127.6.234.195.keytab.
I20260620 11:46:17.398454 7083 external_mini_cluster.cc:1326] Running /tmp/dist-test-task1JRg7N/build/release/bin/kudu
/tmp/dist-test-task1JRg7N/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task1JRg7N/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781955746769051-7083-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-task1JRg7N/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781955746769051-7083-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task1JRg7N/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781955746769051-7083-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-task1JRg7N/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781955746769051-7083-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.6.234.195:0
--local_ip_for_outbound_sockets=127.6.234.195
--webserver_interface=127.6.234.195
--webserver_port=0
--tserver_master_addrs=127.6.234.254:35111
--keytab_file=/tmp/dist-test-task1JRg7N/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781955746769051-7083-0/krb5kdc/kudu_127.6.234.195.keytab
--principal=kudu/127.6.234.195
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.6.234.212:36351
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task1JRg7N/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781955746769051-7083-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task1JRg7N/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781955746769051-7083-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task1JRg7N/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781955746769051-7083-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 11:46:17.480755 25453 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 11:46:17.480968 25453 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 11:46:17.480993 25453 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 11:46:17.482178 25453 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 11:46:17.482219 25453 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 11:46:17.482246 25453 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.6.234.195
I20260620 11:46:17.483709 25453 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.6.234.212:36351
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task1JRg7N/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781955746769051-7083-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-task1JRg7N/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781955746769051-7083-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task1JRg7N/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781955746769051-7083-0/krb5kdc/kudu_127.6.234.195.keytab
--principal=kudu/127.6.234.195
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.6.234.195:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task1JRg7N/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781955746769051-7083-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.6.234.195
--webserver_port=0
--tserver_master_addrs=127.6.234.254:35111
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.25453
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.6.234.195
--log_dir=/tmp/dist-test-task1JRg7N/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781955746769051-7083-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 20 Jun 2026 10:32:37 UTC on ccycloud.quasar-bgyofv.root.comops.site
build id 4457173
I20260620 11:46:17.483897 25453 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 11:46:17.484071 25453 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 11:46:17.489166 25459 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 11:46:17.489195 25458 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 11:46:17.489215 25461 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 11:46:17.489428 25453 server_base.cc:1034] running on GCE node
I20260620 11:46:17.489637 25453 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 11:46:17.489913 25453 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 11:46:17.491068 25453 hybrid_clock.cc:648] HybridClock initialized: now 1781955977491051 us; error 41 us; skew 500 ppm
Jun 20 11:46:17 dist-test-slave-cjc6 krb5kdc[25078](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781955977, etypes {rep=17 tkt=17 ses=17}, kudu/127.6.234.195@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 11:46:17.493850 25453 init.cc:377] Logged in from keytab as kudu/127.6.234.195@KRBTEST.COM (short username kudu)
I20260620 11:46:17.494364 25453 webserver.cc:460] Webserver started at http://127.6.234.195:42467/ using document root <none> and password file <none>
I20260620 11:46:17.494593 25453 fs_manager.cc:359] Metadata directory not provided
I20260620 11:46:17.494661 25453 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 11:46:17.494796 25453 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 11:46:17.495839 25453 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task1JRg7N/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781955746769051-7083-0/minicluster-data/ts-2/data/instance:
uuid: "f632aec6bc874de9ac4fc4ad595cc498"
format_stamp: "Formatted at 2026-06-20 11:46:17 on dist-test-slave-cjc6"
server_key: "713cdb1d6c229a49890ed6947540c708"
server_key_iv: "e4ae70b73ab6eb10f9b70a73fe425991"
server_key_version: "encryptionkey@0"
I20260620 11:46:17.496212 25453 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task1JRg7N/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781955746769051-7083-0/minicluster-data/ts-2/wal/instance:
uuid: "f632aec6bc874de9ac4fc4ad595cc498"
format_stamp: "Formatted at 2026-06-20 11:46:17 on dist-test-slave-cjc6"
server_key: "713cdb1d6c229a49890ed6947540c708"
server_key_iv: "e4ae70b73ab6eb10f9b70a73fe425991"
server_key_version: "encryptionkey@0"
I20260620 11:46:17.497534 25453 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260620 11:46:17.498345 25468 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 11:46:17.498521 25453 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.001s
I20260620 11:46:17.498618 25453 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task1JRg7N/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781955746769051-7083-0/minicluster-data/ts-2/data,/tmp/dist-test-task1JRg7N/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781955746769051-7083-0/minicluster-data/ts-2/wal
uuid: "f632aec6bc874de9ac4fc4ad595cc498"
format_stamp: "Formatted at 2026-06-20 11:46:17 on dist-test-slave-cjc6"
server_key: "713cdb1d6c229a49890ed6947540c708"
server_key_iv: "e4ae70b73ab6eb10f9b70a73fe425991"
server_key_version: "encryptionkey@0"
I20260620 11:46:17.498703 25453 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task1JRg7N/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781955746769051-7083-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-task1JRg7N/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781955746769051-7083-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-task1JRg7N/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781955746769051-7083-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 11:46:17.527390 25453 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 11:46:17.529335 25453 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 11:46:17.529498 25453 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 11:46:17.529798 25453 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 11:46:17.530118 25453 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 11:46:17.530174 25453 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 11:46:17.530215 25453 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 11:46:17.530242 25453 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 11:46:17.536665 25453 rpc_server.cc:307] RPC server started. Bound to: 127.6.234.195:41325
I20260620 11:46:17.537110 25453 server_base.cc:1166] Dumped server information to /tmp/dist-test-task1JRg7N/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781955746769051-7083-0/minicluster-data/ts-2/data/info.pb
I20260620 11:46:17.537885 25581 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.6.234.195:41325 every 8 connection(s)
Jun 20 11:46:17 dist-test-slave-cjc6 krb5kdc[25078](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781955977, etypes {rep=17 tkt=17 ses=17}, kudu/127.6.234.195@KRBTEST.COM for kudu/127.6.234.254@KRBTEST.COM
I20260620 11:46:17.543529 7083 external_mini_cluster.cc:1388] Started /tmp/dist-test-task1JRg7N/build/release/bin/kudu as pid 25453
I20260620 11:46:17.543634 7083 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task1JRg7N/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781955746769051-7083-0/minicluster-data/ts-2/wal/instance
I20260620 11:46:17.543816 7083 external_mini_cluster.cc:1428] Setting key 5b16f1374608b063a324fcbe5f6aed22
I20260620 11:46:17.546212 25162 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 11:46:17.538226 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.6.234.195:52407 (local address 127.6.234.254:35111)
0620 11:46:17.538287 (+ 61us) server_negotiation.cc:206] Beginning negotiation
0620 11:46:17.538289 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 11:46:17.538301 (+ 12us) server_negotiation.cc:407] Connection header received
0620 11:46:17.539171 (+ 870us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 11:46:17.539172 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 11:46:17.539182 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 11:46:17.539243 (+ 61us) server_negotiation.cc:226] Negotiated authn=SASL
0620 11:46:17.539846 (+ 603us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 11:46:17.540342 (+ 496us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 11:46:17.541321 (+ 979us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 11:46:17.541502 (+ 181us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 11:46:17.541556 (+ 54us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 11:46:17.543621 (+ 2065us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 11:46:17.543622 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 11:46:17.543623 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 11:46:17.543632 (+ 9us) server_negotiation.cc:922] calling sasl_server_start()
0620 11:46:17.545089 (+ 1457us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 11:46:17.545439 (+ 350us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 11:46:17.545440 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 11:46:17.545440 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 11:46:17.545494 (+ 54us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 11:46:17.545666 (+ 172us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 11:46:17.545667 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 11:46:17.545667 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 11:46:17.545811 (+ 144us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 11:46:17.545864 (+ 53us) server_negotiation.cc:1035] Waiting for connection context
0620 11:46:17.546095 (+ 231us) server_negotiation.cc:299] Negotiation successful
0620 11:46:17.546152 (+ 57us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":38}
I20260620 11:46:17.546376 25584 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 11:46:17.537907 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.234.254:35111 (local address 127.6.234.195:52407)
0620 11:46:17.538084 (+ 177us) negotiation.cc:107] Waiting for socket to connect
0620 11:46:17.538126 (+ 42us) client_negotiation.cc:174] Beginning negotiation
0620 11:46:17.539079 (+ 953us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 11:46:17.539303 (+ 224us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 11:46:17.539306 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 11:46:17.539556 (+ 250us) client_negotiation.cc:189] Negotiated authn=SASL
0620 11:46:17.539745 (+ 189us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 11:46:17.539749 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 11:46:17.540432 (+ 683us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 11:46:17.540433 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 11:46:17.541251 (+ 818us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 11:46:17.541253 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 11:46:17.541572 (+ 319us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 11:46:17.541573 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 11:46:17.541667 (+ 94us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 11:46:17.542171 (+ 504us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 11:46:17.542179 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0620 11:46:17.543526 (+ 1347us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 11:46:17.545157 (+ 1631us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 11:46:17.545162 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 11:46:17.545164 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0620 11:46:17.545377 (+ 213us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 11:46:17.545536 (+ 159us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 11:46:17.545536 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 11:46:17.545536 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0620 11:46:17.545619 (+ 83us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 11:46:17.545868 (+ 249us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 11:46:17.545873 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 11:46:17.545983 (+ 110us) client_negotiation.cc:769] Sending connection context
0620 11:46:17.546070 (+ 87us) client_negotiation.cc:240] Negotiation successful
0620 11:46:17.546184 (+ 114us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":133,"thread_start_us":104,"threads_started":1}
I20260620 11:46:17.546810 25582 heartbeater.cc:347] Connected to a master server at 127.6.234.254:35111
I20260620 11:46:17.546916 25582 heartbeater.cc:464] Registering TS with master...
I20260620 11:46:17.547086 25582 heartbeater.cc:511] Master 127.6.234.254:35111 requested a full tablet report, sending...
I20260620 11:46:17.547411 25123 ts_manager.cc:194] Registered new tserver with Master: f632aec6bc874de9ac4fc4ad595cc498 (127.6.234.195:41325)
I20260620 11:46:17.547772 25123 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.6.234.195@KRBTEST.COM'} at 127.6.234.195:52407
I20260620 11:46:17.555398 7083 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260620 11:46:17.562999 25162 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 11:46:17.556755 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56932 (local address 127.6.234.254:35111)
0620 11:46:17.556843 (+ 88us) server_negotiation.cc:206] Beginning negotiation
0620 11:46:17.556846 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0620 11:46:17.556928 (+ 82us) server_negotiation.cc:407] Connection header received
0620 11:46:17.557033 (+ 105us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 11:46:17.557033 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 11:46:17.557044 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 11:46:17.557093 (+ 49us) server_negotiation.cc:226] Negotiated authn=SASL
0620 11:46:17.557688 (+ 595us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 11:46:17.558200 (+ 512us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 11:46:17.558951 (+ 751us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 11:46:17.559157 (+ 206us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 11:46:17.559232 (+ 75us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 11:46:17.560518 (+ 1286us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 11:46:17.560520 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 11:46:17.560520 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 11:46:17.560527 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0620 11:46:17.561683 (+ 1156us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 11:46:17.562147 (+ 464us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 11:46:17.562148 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 11:46:17.562148 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 11:46:17.562211 (+ 63us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 11:46:17.562436 (+ 225us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 11:46:17.562437 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 11:46:17.562437 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 11:46:17.562580 (+ 143us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 11:46:17.562713 (+ 133us) server_negotiation.cc:1035] Waiting for connection context
0620 11:46:17.562825 (+ 112us) server_negotiation.cc:299] Negotiation successful
0620 11:46:17.562901 (+ 76us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":60}
I20260620 11:46:17.564689 25123 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:56932:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260620 11:46:17.566460 25123 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260620 11:46:17.573124 25596 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 11:46:17.569044 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60450 (local address 127.6.234.194:46385)
0620 11:46:17.569202 (+ 158us) server_negotiation.cc:206] Beginning negotiation
0620 11:46:17.569207 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 11:46:17.569336 (+ 129us) server_negotiation.cc:407] Connection header received
0620 11:46:17.569380 (+ 44us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 11:46:17.569382 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 11:46:17.569395 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 11:46:17.569469 (+ 74us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 11:46:17.569727 (+ 258us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 11:46:17.570846 (+ 1119us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 11:46:17.572125 (+ 1279us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 11:46:17.572787 (+ 662us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 11:46:17.572850 (+ 63us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 11:46:17.572897 (+ 47us) server_negotiation.cc:1035] Waiting for connection context
0620 11:46:17.573008 (+ 111us) server_negotiation.cc:299] Negotiation successful
0620 11:46:17.573047 (+ 39us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":125,"thread_start_us":87,"threads_started":1}
I20260620 11:46:17.573166 25595 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 11:46:17.568791 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.234.194:46385 (local address 127.0.0.1:60450)
0620 11:46:17.569217 (+ 426us) negotiation.cc:107] Waiting for socket to connect
0620 11:46:17.569229 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0620 11:46:17.569281 (+ 52us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 11:46:17.569549 (+ 268us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 11:46:17.569550 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 11:46:17.569553 (+ 3us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 11:46:17.569625 (+ 72us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 11:46:17.569628 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 11:46:17.570948 (+ 1320us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 11:46:17.570949 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 11:46:17.572053 (+ 1104us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 11:46:17.572054 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 11:46:17.572858 (+ 804us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 11:46:17.572859 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 11:46:17.572906 (+ 47us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 11:46:17.572922 (+ 16us) client_negotiation.cc:769] Sending connection context
0620 11:46:17.572998 (+ 76us) client_negotiation.cc:240] Negotiation successful
0620 11:46:17.573070 (+ 72us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":407,"mutex_wait_us":32,"thread_start_us":91,"threads_started":1}
I20260620 11:46:17.573166 25594 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 11:46:17.568787 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.234.193:38993 (local address 127.0.0.1:36006)
0620 11:46:17.568930 (+ 143us) negotiation.cc:107] Waiting for socket to connect
0620 11:46:17.568972 (+ 42us) client_negotiation.cc:174] Beginning negotiation
0620 11:46:17.569045 (+ 73us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 11:46:17.569437 (+ 392us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 11:46:17.569440 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 11:46:17.569442 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 11:46:17.569514 (+ 72us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 11:46:17.569516 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 11:46:17.570603 (+ 1087us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 11:46:17.570604 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 11:46:17.571875 (+ 1271us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 11:46:17.571878 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 11:46:17.572808 (+ 930us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 11:46:17.572809 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 11:46:17.572906 (+ 97us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 11:46:17.572922 (+ 16us) client_negotiation.cc:769] Sending connection context
0620 11:46:17.572998 (+ 76us) client_negotiation.cc:240] Negotiation successful
0620 11:46:17.573072 (+ 74us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":112,"mutex_wait_us":33,"thread_start_us":57,"threads_started":1}
I20260620 11:46:17.573675 25598 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 11:46:17.569077 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36006 (local address 127.6.234.193:38993)
0620 11:46:17.569251 (+ 174us) server_negotiation.cc:206] Beginning negotiation
0620 11:46:17.569256 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 11:46:17.569269 (+ 13us) server_negotiation.cc:407] Connection header received
0620 11:46:17.569308 (+ 39us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 11:46:17.569311 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 11:46:17.569327 (+ 16us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 11:46:17.569429 (+ 102us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 11:46:17.569644 (+ 215us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 11:46:17.570492 (+ 848us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 11:46:17.571970 (+ 1478us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 11:46:17.572723 (+ 753us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 11:46:17.572807 (+ 84us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 11:46:17.572874 (+ 67us) server_negotiation.cc:1035] Waiting for connection context
0620 11:46:17.573545 (+ 671us) server_negotiation.cc:299] Negotiation successful
0620 11:46:17.573593 (+ 48us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":140,"thread_start_us":75,"threads_started":1}
I20260620 11:46:17.574108 25593 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 11:46:17.568824 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.234.195:41325 (local address 127.0.0.1:43934)
0620 11:46:17.569238 (+ 414us) negotiation.cc:107] Waiting for socket to connect
0620 11:46:17.569250 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0620 11:46:17.569304 (+ 54us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 11:46:17.569549 (+ 245us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 11:46:17.569550 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 11:46:17.569556 (+ 6us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 11:46:17.569625 (+ 69us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 11:46:17.569628 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 11:46:17.570963 (+ 1335us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 11:46:17.570964 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 11:46:17.571876 (+ 912us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 11:46:17.571885 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 11:46:17.573750 (+ 1865us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 11:46:17.573752 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 11:46:17.573925 (+ 173us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 11:46:17.573930 (+ 5us) client_negotiation.cc:769] Sending connection context
0620 11:46:17.574001 (+ 71us) client_negotiation.cc:240] Negotiation successful
0620 11:46:17.574046 (+ 45us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":400,"mutex_wait_us":38,"thread_start_us":68,"threads_started":1}
I20260620 11:46:17.574110 25597 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 11:46:17.569034 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43934 (local address 127.6.234.195:41325)
0620 11:46:17.569290 (+ 256us) server_negotiation.cc:206] Beginning negotiation
0620 11:46:17.569294 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 11:46:17.569305 (+ 11us) server_negotiation.cc:407] Connection header received
0620 11:46:17.569377 (+ 72us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 11:46:17.569379 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 11:46:17.569395 (+ 16us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 11:46:17.569469 (+ 74us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 11:46:17.569728 (+ 259us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 11:46:17.570847 (+ 1119us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 11:46:17.572914 (+ 2067us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 11:46:17.573641 (+ 727us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 11:46:17.573749 (+ 108us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 11:46:17.573818 (+ 69us) server_negotiation.cc:1035] Waiting for connection context
0620 11:46:17.574016 (+ 198us) server_negotiation.cc:299] Negotiation successful
0620 11:46:17.574049 (+ 33us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":223,"thread_start_us":103,"threads_started":1}
I20260620 11:46:17.574010 25382 tablet_service.cc:1467] Processing CreateTablet for tablet 73233d369a6b4370b74119c49e73f0f4 (DEFAULT_TABLE table=test-table [id=a1c01c5e2b1742ecaa5dbb024d0f9557]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 11:46:17.574397 25382 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 73233d369a6b4370b74119c49e73f0f4. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 11:46:17.576987 25599 tablet_bootstrap.cc:492] T 73233d369a6b4370b74119c49e73f0f4 P 789253646a8442f985c63e3d47306976: Bootstrap starting.
I20260620 11:46:17.577401 25516 tablet_service.cc:1467] Processing CreateTablet for tablet 73233d369a6b4370b74119c49e73f0f4 (DEFAULT_TABLE table=test-table [id=a1c01c5e2b1742ecaa5dbb024d0f9557]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 11:46:17.577621 25599 tablet_bootstrap.cc:654] T 73233d369a6b4370b74119c49e73f0f4 P 789253646a8442f985c63e3d47306976: Neither blocks nor log segments found. Creating new log.
I20260620 11:46:17.577745 25516 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 73233d369a6b4370b74119c49e73f0f4. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 11:46:17.578189 25599 log.cc:824] T 73233d369a6b4370b74119c49e73f0f4 P 789253646a8442f985c63e3d47306976: Log is configured to *not* fsync() on all Append() calls
I20260620 11:46:17.578917 25599 tablet_bootstrap.cc:492] T 73233d369a6b4370b74119c49e73f0f4 P 789253646a8442f985c63e3d47306976: No bootstrap required, opened a new log
I20260620 11:46:17.578971 25599 ts_tablet_manager.cc:1397] T 73233d369a6b4370b74119c49e73f0f4 P 789253646a8442f985c63e3d47306976: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20260620 11:46:17.580284 25601 tablet_bootstrap.cc:492] T 73233d369a6b4370b74119c49e73f0f4 P f632aec6bc874de9ac4fc4ad595cc498: Bootstrap starting.
I20260620 11:46:17.580331 25599 raft_consensus.cc:348] T 73233d369a6b4370b74119c49e73f0f4 P 789253646a8442f985c63e3d47306976 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "65d7c74f35e648e181d1eb2f8419924e" member_type: VOTER last_known_addr { host: "127.6.234.193" port: 38993 } } peers { permanent_uuid: "789253646a8442f985c63e3d47306976" member_type: VOTER last_known_addr { host: "127.6.234.194" port: 46385 } } peers { permanent_uuid: "f632aec6bc874de9ac4fc4ad595cc498" member_type: VOTER last_known_addr { host: "127.6.234.195" port: 41325 } }
I20260620 11:46:17.580710 25599 raft_consensus.cc:374] T 73233d369a6b4370b74119c49e73f0f4 P 789253646a8442f985c63e3d47306976 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 11:46:17.580781 25599 raft_consensus.cc:729] T 73233d369a6b4370b74119c49e73f0f4 P 789253646a8442f985c63e3d47306976 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 789253646a8442f985c63e3d47306976, State: Initialized, Role: FOLLOWER
I20260620 11:46:17.580915 25599 consensus_queue.cc:260] T 73233d369a6b4370b74119c49e73f0f4 P 789253646a8442f985c63e3d47306976 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "65d7c74f35e648e181d1eb2f8419924e" member_type: VOTER last_known_addr { host: "127.6.234.193" port: 38993 } } peers { permanent_uuid: "789253646a8442f985c63e3d47306976" member_type: VOTER last_known_addr { host: "127.6.234.194" port: 46385 } } peers { permanent_uuid: "f632aec6bc874de9ac4fc4ad595cc498" member_type: VOTER last_known_addr { host: "127.6.234.195" port: 41325 } }
I20260620 11:46:17.581058 25601 tablet_bootstrap.cc:654] T 73233d369a6b4370b74119c49e73f0f4 P f632aec6bc874de9ac4fc4ad595cc498: Neither blocks nor log segments found. Creating new log.
I20260620 11:46:17.581192 25599 ts_tablet_manager.cc:1428] T 73233d369a6b4370b74119c49e73f0f4 P 789253646a8442f985c63e3d47306976: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20260620 11:46:17.581138 25248 tablet_service.cc:1467] Processing CreateTablet for tablet 73233d369a6b4370b74119c49e73f0f4 (DEFAULT_TABLE table=test-table [id=a1c01c5e2b1742ecaa5dbb024d0f9557]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 11:46:17.581477 25248 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 73233d369a6b4370b74119c49e73f0f4. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 11:46:17.581637 25601 log.cc:824] T 73233d369a6b4370b74119c49e73f0f4 P f632aec6bc874de9ac4fc4ad595cc498: Log is configured to *not* fsync() on all Append() calls
I20260620 11:46:17.581804 25448 heartbeater.cc:503] Master 127.6.234.254:35111 was elected leader, sending a full tablet report...
I20260620 11:46:17.582415 25601 tablet_bootstrap.cc:492] T 73233d369a6b4370b74119c49e73f0f4 P f632aec6bc874de9ac4fc4ad595cc498: No bootstrap required, opened a new log
I20260620 11:46:17.582495 25601 ts_tablet_manager.cc:1397] T 73233d369a6b4370b74119c49e73f0f4 P f632aec6bc874de9ac4fc4ad595cc498: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20260620 11:46:17.584494 25601 raft_consensus.cc:348] T 73233d369a6b4370b74119c49e73f0f4 P f632aec6bc874de9ac4fc4ad595cc498 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "65d7c74f35e648e181d1eb2f8419924e" member_type: VOTER last_known_addr { host: "127.6.234.193" port: 38993 } } peers { permanent_uuid: "789253646a8442f985c63e3d47306976" member_type: VOTER last_known_addr { host: "127.6.234.194" port: 46385 } } peers { permanent_uuid: "f632aec6bc874de9ac4fc4ad595cc498" member_type: VOTER last_known_addr { host: "127.6.234.195" port: 41325 } }
I20260620 11:46:17.584674 25601 raft_consensus.cc:374] T 73233d369a6b4370b74119c49e73f0f4 P f632aec6bc874de9ac4fc4ad595cc498 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 11:46:17.584717 25601 raft_consensus.cc:729] T 73233d369a6b4370b74119c49e73f0f4 P f632aec6bc874de9ac4fc4ad595cc498 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f632aec6bc874de9ac4fc4ad595cc498, State: Initialized, Role: FOLLOWER
I20260620 11:46:17.584730 25604 tablet_bootstrap.cc:492] T 73233d369a6b4370b74119c49e73f0f4 P 65d7c74f35e648e181d1eb2f8419924e: Bootstrap starting.
I20260620 11:46:17.584834 25601 consensus_queue.cc:260] T 73233d369a6b4370b74119c49e73f0f4 P f632aec6bc874de9ac4fc4ad595cc498 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "65d7c74f35e648e181d1eb2f8419924e" member_type: VOTER last_known_addr { host: "127.6.234.193" port: 38993 } } peers { permanent_uuid: "789253646a8442f985c63e3d47306976" member_type: VOTER last_known_addr { host: "127.6.234.194" port: 46385 } } peers { permanent_uuid: "f632aec6bc874de9ac4fc4ad595cc498" member_type: VOTER last_known_addr { host: "127.6.234.195" port: 41325 } }
I20260620 11:46:17.585126 25601 ts_tablet_manager.cc:1428] T 73233d369a6b4370b74119c49e73f0f4 P f632aec6bc874de9ac4fc4ad595cc498: Time spent starting tablet: real 0.003s user 0.003s sys 0.000s
I20260620 11:46:17.585165 25582 heartbeater.cc:503] Master 127.6.234.254:35111 was elected leader, sending a full tablet report...
I20260620 11:46:17.585448 25604 tablet_bootstrap.cc:654] T 73233d369a6b4370b74119c49e73f0f4 P 65d7c74f35e648e181d1eb2f8419924e: Neither blocks nor log segments found. Creating new log.
I20260620 11:46:17.586076 25604 log.cc:824] T 73233d369a6b4370b74119c49e73f0f4 P 65d7c74f35e648e181d1eb2f8419924e: Log is configured to *not* fsync() on all Append() calls
I20260620 11:46:17.586827 25604 tablet_bootstrap.cc:492] T 73233d369a6b4370b74119c49e73f0f4 P 65d7c74f35e648e181d1eb2f8419924e: No bootstrap required, opened a new log
I20260620 11:46:17.586894 25604 ts_tablet_manager.cc:1397] T 73233d369a6b4370b74119c49e73f0f4 P 65d7c74f35e648e181d1eb2f8419924e: Time spent bootstrapping tablet: real 0.002s user 0.000s sys 0.001s
I20260620 11:46:17.588245 25604 raft_consensus.cc:348] T 73233d369a6b4370b74119c49e73f0f4 P 65d7c74f35e648e181d1eb2f8419924e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "65d7c74f35e648e181d1eb2f8419924e" member_type: VOTER last_known_addr { host: "127.6.234.193" port: 38993 } } peers { permanent_uuid: "789253646a8442f985c63e3d47306976" member_type: VOTER last_known_addr { host: "127.6.234.194" port: 46385 } } peers { permanent_uuid: "f632aec6bc874de9ac4fc4ad595cc498" member_type: VOTER last_known_addr { host: "127.6.234.195" port: 41325 } }
I20260620 11:46:17.588395 25604 raft_consensus.cc:374] T 73233d369a6b4370b74119c49e73f0f4 P 65d7c74f35e648e181d1eb2f8419924e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 11:46:17.588441 25604 raft_consensus.cc:729] T 73233d369a6b4370b74119c49e73f0f4 P 65d7c74f35e648e181d1eb2f8419924e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 65d7c74f35e648e181d1eb2f8419924e, State: Initialized, Role: FOLLOWER
I20260620 11:46:17.588563 25604 consensus_queue.cc:260] T 73233d369a6b4370b74119c49e73f0f4 P 65d7c74f35e648e181d1eb2f8419924e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "65d7c74f35e648e181d1eb2f8419924e" member_type: VOTER last_known_addr { host: "127.6.234.193" port: 38993 } } peers { permanent_uuid: "789253646a8442f985c63e3d47306976" member_type: VOTER last_known_addr { host: "127.6.234.194" port: 46385 } } peers { permanent_uuid: "f632aec6bc874de9ac4fc4ad595cc498" member_type: VOTER last_known_addr { host: "127.6.234.195" port: 41325 } }
I20260620 11:46:17.588812 25604 ts_tablet_manager.cc:1428] T 73233d369a6b4370b74119c49e73f0f4 P 65d7c74f35e648e181d1eb2f8419924e: Time spent starting tablet: real 0.002s user 0.000s sys 0.002s
I20260620 11:46:17.588857 25314 heartbeater.cc:503] Master 127.6.234.254:35111 was elected leader, sending a full tablet report...
W20260620 11:46:17.614176 25449 tablet.cc:2369] T 73233d369a6b4370b74119c49e73f0f4 P 789253646a8442f985c63e3d47306976: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260620 11:46:17.615067 25607 raft_consensus.cc:482] T 73233d369a6b4370b74119c49e73f0f4 P 65d7c74f35e648e181d1eb2f8419924e [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260620 11:46:17.615159 25607 raft_consensus.cc:504] T 73233d369a6b4370b74119c49e73f0f4 P 65d7c74f35e648e181d1eb2f8419924e [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "65d7c74f35e648e181d1eb2f8419924e" member_type: VOTER last_known_addr { host: "127.6.234.193" port: 38993 } } peers { permanent_uuid: "789253646a8442f985c63e3d47306976" member_type: VOTER last_known_addr { host: "127.6.234.194" port: 46385 } } peers { permanent_uuid: "f632aec6bc874de9ac4fc4ad595cc498" member_type: VOTER last_known_addr { host: "127.6.234.195" port: 41325 } }
I20260620 11:46:17.615509 25607 leader_election.cc:288] T 73233d369a6b4370b74119c49e73f0f4 P 65d7c74f35e648e181d1eb2f8419924e [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 789253646a8442f985c63e3d47306976 (127.6.234.194:46385), f632aec6bc874de9ac4fc4ad595cc498 (127.6.234.195:41325)
I20260620 11:46:17.619401 25596 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 11:46:17.615713 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.6.234.193:57397 (local address 127.6.234.194:46385)
0620 11:46:17.615757 (+ 44us) server_negotiation.cc:206] Beginning negotiation
0620 11:46:17.615759 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 11:46:17.615769 (+ 10us) server_negotiation.cc:407] Connection header received
0620 11:46:17.615864 (+ 95us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 11:46:17.615865 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 11:46:17.615877 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 11:46:17.615944 (+ 67us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 11:46:17.616136 (+ 192us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 11:46:17.616908 (+ 772us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 11:46:17.618451 (+ 1543us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 11:46:17.619035 (+ 584us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 11:46:17.619118 (+ 83us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 11:46:17.619126 (+ 8us) server_negotiation.cc:1035] Waiting for connection context
0620 11:46:17.619294 (+ 168us) server_negotiation.cc:299] Negotiation successful
0620 11:46:17.619333 (+ 39us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":23}
I20260620 11:46:17.619401 25597 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 11:46:17.615752 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.6.234.193:39473 (local address 127.6.234.195:41325)
0620 11:46:17.615800 (+ 48us) server_negotiation.cc:206] Beginning negotiation
0620 11:46:17.615802 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 11:46:17.615946 (+ 144us) server_negotiation.cc:407] Connection header received
0620 11:46:17.615988 (+ 42us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 11:46:17.615989 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 11:46:17.615999 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 11:46:17.616047 (+ 48us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 11:46:17.616199 (+ 152us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 11:46:17.616907 (+ 708us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 11:46:17.618452 (+ 1545us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 11:46:17.619055 (+ 603us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 11:46:17.619117 (+ 62us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 11:46:17.619126 (+ 9us) server_negotiation.cc:1035] Waiting for connection context
0620 11:46:17.619294 (+ 168us) server_negotiation.cc:299] Negotiation successful
0620 11:46:17.619333 (+ 39us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":23}
I20260620 11:46:17.619426 25316 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 11:46:17.615651 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.234.194:46385 (local address 127.6.234.193:57397)
0620 11:46:17.615709 (+ 58us) negotiation.cc:107] Waiting for socket to connect
0620 11:46:17.615722 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0620 11:46:17.615764 (+ 42us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 11:46:17.615959 (+ 195us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 11:46:17.615960 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 11:46:17.615962 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 11:46:17.616063 (+ 101us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 11:46:17.616066 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 11:46:17.616991 (+ 925us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 11:46:17.616992 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 11:46:17.618360 (+ 1368us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 11:46:17.618364 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 11:46:17.619128 (+ 764us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 11:46:17.619129 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 11:46:17.619212 (+ 83us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 11:46:17.619214 (+ 2us) client_negotiation.cc:769] Sending connection context
0620 11:46:17.619270 (+ 56us) client_negotiation.cc:240] Negotiation successful
0620 11:46:17.619321 (+ 51us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":37,"mutex_wait_us":68}
I20260620 11:46:17.619431 25608 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 11:46:17.615673 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.234.195:41325 (local address 127.6.234.193:39473)
0620 11:46:17.615868 (+ 195us) negotiation.cc:107] Waiting for socket to connect
0620 11:46:17.615877 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0620 11:46:17.615937 (+ 60us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 11:46:17.616065 (+ 128us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 11:46:17.616066 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 11:46:17.616068 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 11:46:17.616131 (+ 63us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 11:46:17.616132 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 11:46:17.616991 (+ 859us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 11:46:17.616992 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 11:46:17.618360 (+ 1368us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 11:46:17.618364 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 11:46:17.619128 (+ 764us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 11:46:17.619129 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 11:46:17.619208 (+ 79us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 11:46:17.619212 (+ 4us) client_negotiation.cc:769] Sending connection context
0620 11:46:17.619270 (+ 58us) client_negotiation.cc:240] Negotiation successful
0620 11:46:17.619320 (+ 50us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":180,"mutex_wait_us":97,"thread_start_us":122,"threads_started":1}
I20260620 11:46:17.619791 25402 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "73233d369a6b4370b74119c49e73f0f4" candidate_uuid: "65d7c74f35e648e181d1eb2f8419924e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "789253646a8442f985c63e3d47306976" is_pre_election: true
I20260620 11:46:17.619972 25402 raft_consensus.cc:2454] T 73233d369a6b4370b74119c49e73f0f4 P 789253646a8442f985c63e3d47306976 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 65d7c74f35e648e181d1eb2f8419924e in term 0.
I20260620 11:46:17.620121 25536 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "73233d369a6b4370b74119c49e73f0f4" candidate_uuid: "65d7c74f35e648e181d1eb2f8419924e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f632aec6bc874de9ac4fc4ad595cc498" is_pre_election: true
I20260620 11:46:17.620199 25201 leader_election.cc:302] T 73233d369a6b4370b74119c49e73f0f4 P 65d7c74f35e648e181d1eb2f8419924e [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 65d7c74f35e648e181d1eb2f8419924e, 789253646a8442f985c63e3d47306976; no voters:
I20260620 11:46:17.620287 25536 raft_consensus.cc:2454] T 73233d369a6b4370b74119c49e73f0f4 P f632aec6bc874de9ac4fc4ad595cc498 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 65d7c74f35e648e181d1eb2f8419924e in term 0.
I20260620 11:46:17.620371 25607 raft_consensus.cc:2781] T 73233d369a6b4370b74119c49e73f0f4 P 65d7c74f35e648e181d1eb2f8419924e [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260620 11:46:17.620453 25607 raft_consensus.cc:482] T 73233d369a6b4370b74119c49e73f0f4 P 65d7c74f35e648e181d1eb2f8419924e [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260620 11:46:17.620532 25607 raft_consensus.cc:3037] T 73233d369a6b4370b74119c49e73f0f4 P 65d7c74f35e648e181d1eb2f8419924e [term 0 FOLLOWER]: Advancing to term 1
I20260620 11:46:17.621260 25607 raft_consensus.cc:504] T 73233d369a6b4370b74119c49e73f0f4 P 65d7c74f35e648e181d1eb2f8419924e [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "65d7c74f35e648e181d1eb2f8419924e" member_type: VOTER last_known_addr { host: "127.6.234.193" port: 38993 } } peers { permanent_uuid: "789253646a8442f985c63e3d47306976" member_type: VOTER last_known_addr { host: "127.6.234.194" port: 46385 } } peers { permanent_uuid: "f632aec6bc874de9ac4fc4ad595cc498" member_type: VOTER last_known_addr { host: "127.6.234.195" port: 41325 } }
I20260620 11:46:17.621408 25607 leader_election.cc:288] T 73233d369a6b4370b74119c49e73f0f4 P 65d7c74f35e648e181d1eb2f8419924e [CANDIDATE]: Term 1 election: Requested vote from peers 789253646a8442f985c63e3d47306976 (127.6.234.194:46385), f632aec6bc874de9ac4fc4ad595cc498 (127.6.234.195:41325)
I20260620 11:46:17.621560 25536 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "73233d369a6b4370b74119c49e73f0f4" candidate_uuid: "65d7c74f35e648e181d1eb2f8419924e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f632aec6bc874de9ac4fc4ad595cc498"
I20260620 11:46:17.621575 25402 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "73233d369a6b4370b74119c49e73f0f4" candidate_uuid: "65d7c74f35e648e181d1eb2f8419924e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "789253646a8442f985c63e3d47306976"
I20260620 11:46:17.621627 25536 raft_consensus.cc:3037] T 73233d369a6b4370b74119c49e73f0f4 P f632aec6bc874de9ac4fc4ad595cc498 [term 0 FOLLOWER]: Advancing to term 1
I20260620 11:46:17.621687 25402 raft_consensus.cc:3037] T 73233d369a6b4370b74119c49e73f0f4 P 789253646a8442f985c63e3d47306976 [term 0 FOLLOWER]: Advancing to term 1
I20260620 11:46:17.622368 25536 raft_consensus.cc:2454] T 73233d369a6b4370b74119c49e73f0f4 P f632aec6bc874de9ac4fc4ad595cc498 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 65d7c74f35e648e181d1eb2f8419924e in term 1.
I20260620 11:46:17.622463 25402 raft_consensus.cc:2454] T 73233d369a6b4370b74119c49e73f0f4 P 789253646a8442f985c63e3d47306976 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 65d7c74f35e648e181d1eb2f8419924e in term 1.
I20260620 11:46:17.622558 25203 leader_election.cc:302] T 73233d369a6b4370b74119c49e73f0f4 P 65d7c74f35e648e181d1eb2f8419924e [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 65d7c74f35e648e181d1eb2f8419924e, f632aec6bc874de9ac4fc4ad595cc498; no voters:
I20260620 11:46:17.622664 25607 raft_consensus.cc:2781] T 73233d369a6b4370b74119c49e73f0f4 P 65d7c74f35e648e181d1eb2f8419924e [term 1 FOLLOWER]: Leader election won for term 1
I20260620 11:46:17.622859 25607 raft_consensus.cc:686] T 73233d369a6b4370b74119c49e73f0f4 P 65d7c74f35e648e181d1eb2f8419924e [term 1 LEADER]: Becoming Leader. State: Replica: 65d7c74f35e648e181d1eb2f8419924e, State: Running, Role: LEADER
I20260620 11:46:17.622969 25607 consensus_queue.cc:237] T 73233d369a6b4370b74119c49e73f0f4 P 65d7c74f35e648e181d1eb2f8419924e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "65d7c74f35e648e181d1eb2f8419924e" member_type: VOTER last_known_addr { host: "127.6.234.193" port: 38993 } } peers { permanent_uuid: "789253646a8442f985c63e3d47306976" member_type: VOTER last_known_addr { host: "127.6.234.194" port: 46385 } } peers { permanent_uuid: "f632aec6bc874de9ac4fc4ad595cc498" member_type: VOTER last_known_addr { host: "127.6.234.195" port: 41325 } }
I20260620 11:46:17.623764 25123 catalog_manager.cc:5495] T 73233d369a6b4370b74119c49e73f0f4 P 65d7c74f35e648e181d1eb2f8419924e reported cstate change: term changed from 0 to 1, leader changed from <none> to 65d7c74f35e648e181d1eb2f8419924e (127.6.234.193). New cstate: current_term: 1 leader_uuid: "65d7c74f35e648e181d1eb2f8419924e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "65d7c74f35e648e181d1eb2f8419924e" member_type: VOTER last_known_addr { host: "127.6.234.193" port: 38993 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "789253646a8442f985c63e3d47306976" member_type: VOTER last_known_addr { host: "127.6.234.194" port: 46385 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "f632aec6bc874de9ac4fc4ad595cc498" member_type: VOTER last_known_addr { host: "127.6.234.195" port: 41325 } health_report { overall_health: UNKNOWN } } }
I20260620 11:46:17.636997 25598 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 11:46:17.634512 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36010 (local address 127.6.234.193:38993)
0620 11:46:17.634557 (+ 45us) server_negotiation.cc:206] Beginning negotiation
0620 11:46:17.634559 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 11:46:17.634568 (+ 9us) server_negotiation.cc:407] Connection header received
0620 11:46:17.634661 (+ 93us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 11:46:17.634662 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 11:46:17.634681 (+ 19us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 11:46:17.634759 (+ 78us) server_negotiation.cc:226] Negotiated authn=TOKEN
0620 11:46:17.634919 (+ 160us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 11:46:17.635387 (+ 468us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 11:46:17.636184 (+ 797us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 11:46:17.636379 (+ 195us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 11:46:17.636465 (+ 86us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 11:46:17.636629 (+ 164us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0620 11:46:17.636726 (+ 97us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0620 11:46:17.636836 (+ 110us) server_negotiation.cc:1035] Waiting for connection context
0620 11:46:17.636886 (+ 50us) server_negotiation.cc:299] Negotiation successful
0620 11:46:17.636898 (+ 12us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":22}
I20260620 11:46:17.638759 25402 raft_consensus.cc:1261] T 73233d369a6b4370b74119c49e73f0f4 P 789253646a8442f985c63e3d47306976 [term 1 FOLLOWER]: Refusing update from remote peer 65d7c74f35e648e181d1eb2f8419924e: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 11:46:17.638769 25536 raft_consensus.cc:1261] T 73233d369a6b4370b74119c49e73f0f4 P f632aec6bc874de9ac4fc4ad595cc498 [term 1 FOLLOWER]: Refusing update from remote peer 65d7c74f35e648e181d1eb2f8419924e: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 11:46:17.639129 25607 consensus_queue.cc:1035] T 73233d369a6b4370b74119c49e73f0f4 P 65d7c74f35e648e181d1eb2f8419924e [LEADER]: Connected to new peer: Peer: permanent_uuid: "789253646a8442f985c63e3d47306976" member_type: VOTER last_known_addr { host: "127.6.234.194" port: 46385 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 11:46:17.639287 25609 consensus_queue.cc:1035] T 73233d369a6b4370b74119c49e73f0f4 P 65d7c74f35e648e181d1eb2f8419924e [LEADER]: Connected to new peer: Peer: permanent_uuid: "f632aec6bc874de9ac4fc4ad595cc498" member_type: VOTER last_known_addr { host: "127.6.234.195" port: 41325 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 11:46:17.641708 25613 mvcc.cc:205] Tried to move back new op lower bound from 7298891684405739520 to 7298891684344176640. Current Snapshot: MvccSnapshot[applied={T|T < 7298891684405739520}]
I20260620 11:46:17.641987 25614 mvcc.cc:205] Tried to move back new op lower bound from 7298891684405739520 to 7298891684344176640. Current Snapshot: MvccSnapshot[applied={T|T < 7298891684405739520}]
I20260620 11:46:17.642733 25612 mvcc.cc:205] Tried to move back new op lower bound from 7298891684405739520 to 7298891684344176640. Current Snapshot: MvccSnapshot[applied={T|T < 7298891684405739520 or (T in {7298891684405739520})}]
I20260620 11:46:17.643751 25123 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:56932:
table { table_name: "test-table" } modify_external_catalogs: true
I20260620 11:46:17.643849 25123 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:56932:
table { table_name: "test-table" } modify_external_catalogs: true
I20260620 11:46:17.644557 25123 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 0d852506e1ef4719b025ac0a447a18dd: Sending DeleteTablet for 3 replicas of tablet 73233d369a6b4370b74119c49e73f0f4
I20260620 11:46:17.644883 25248 tablet_service.cc:1514] Processing DeleteTablet for tablet 73233d369a6b4370b74119c49e73f0f4 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 11:46:17 UTC) from {username='kudu', principal='kudu/127.6.234.254@KRBTEST.COM'} at 127.0.0.1:36006
I20260620 11:46:17.644893 25516 tablet_service.cc:1514] Processing DeleteTablet for tablet 73233d369a6b4370b74119c49e73f0f4 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 11:46:17 UTC) from {username='kudu', principal='kudu/127.6.234.254@KRBTEST.COM'} at 127.0.0.1:43934
I20260620 11:46:17.645114 25622 tablet_replica.cc:330] T 73233d369a6b4370b74119c49e73f0f4 P 65d7c74f35e648e181d1eb2f8419924e: stopping tablet replica
I20260620 11:46:17.645210 25622 raft_consensus.cc:2229] T 73233d369a6b4370b74119c49e73f0f4 P 65d7c74f35e648e181d1eb2f8419924e [term 1 LEADER]: Raft consensus shutting down.
I20260620 11:46:17.645327 25622 raft_consensus.cc:2258] T 73233d369a6b4370b74119c49e73f0f4 P 65d7c74f35e648e181d1eb2f8419924e [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 11:46:17.645501 25621 tablet_replica.cc:330] T 73233d369a6b4370b74119c49e73f0f4 P f632aec6bc874de9ac4fc4ad595cc498: stopping tablet replica
I20260620 11:46:17.645613 25621 raft_consensus.cc:2229] T 73233d369a6b4370b74119c49e73f0f4 P f632aec6bc874de9ac4fc4ad595cc498 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260620 11:46:17.645721 7083 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task1JRg7N/build/release/bin/kudu with pid 25185
I20260620 11:46:17.647392 25382 tablet_service.cc:1514] Processing DeleteTablet for tablet 73233d369a6b4370b74119c49e73f0f4 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 11:46:17 UTC) from {username='kudu', principal='kudu/127.6.234.254@KRBTEST.COM'} at 127.0.0.1:60450
I20260620 11:46:17.647625 25623 tablet_replica.cc:330] T 73233d369a6b4370b74119c49e73f0f4 P 789253646a8442f985c63e3d47306976: stopping tablet replica
I20260620 11:46:17.647734 25623 raft_consensus.cc:2229] T 73233d369a6b4370b74119c49e73f0f4 P 789253646a8442f985c63e3d47306976 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260620 11:46:17.647836 25623 raft_consensus.cc:2258] T 73233d369a6b4370b74119c49e73f0f4 P 789253646a8442f985c63e3d47306976 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 11:46:17.648252 25623 ts_tablet_manager.cc:1905] T 73233d369a6b4370b74119c49e73f0f4 P 789253646a8442f985c63e3d47306976: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 11:46:17.649844 25621 raft_consensus.cc:2258] T 73233d369a6b4370b74119c49e73f0f4 P f632aec6bc874de9ac4fc4ad595cc498 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 11:46:17.650261 25621 ts_tablet_manager.cc:1905] T 73233d369a6b4370b74119c49e73f0f4 P f632aec6bc874de9ac4fc4ad595cc498: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 11:46:17.650354 25623 ts_tablet_manager.cc:1918] T 73233d369a6b4370b74119c49e73f0f4 P 789253646a8442f985c63e3d47306976: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260620 11:46:17.650436 25623 log.cc:1196] T 73233d369a6b4370b74119c49e73f0f4 P 789253646a8442f985c63e3d47306976: Deleting WAL directory at /tmp/dist-test-task1JRg7N/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781955746769051-7083-0/minicluster-data/ts-1/wal/wals/73233d369a6b4370b74119c49e73f0f4
I20260620 11:46:17.650704 25623 ts_tablet_manager.cc:1939] T 73233d369a6b4370b74119c49e73f0f4 P 789253646a8442f985c63e3d47306976: Deleting consensus metadata
I20260620 11:46:17.651096 25107 catalog_manager.cc:4841] TS 789253646a8442f985c63e3d47306976 (127.6.234.194:46385): tablet 73233d369a6b4370b74119c49e73f0f4 (table test-table [id=a1c01c5e2b1742ecaa5dbb024d0f9557]) successfully deleted
W20260620 11:46:17.651569 25110 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.6.234.193:38993 (error 108)
I20260620 11:46:17.651811 7083 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task1JRg7N/build/release/bin/kudu with pid 25319
I20260620 11:46:17.652382 25593 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 11:46:17.651918 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.234.193:38993 (local address 127.0.0.1:36012)
0620 11:46:17.652298 (+ 380us) negotiation.cc:107] Waiting for socket to connect
0620 11:46:17.652341 (+ 43us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.234.193:38993: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":358}
I20260620 11:46:17.652639 25621 ts_tablet_manager.cc:1918] T 73233d369a6b4370b74119c49e73f0f4 P f632aec6bc874de9ac4fc4ad595cc498: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260620 11:46:17.652688 25621 log.cc:1196] T 73233d369a6b4370b74119c49e73f0f4 P f632aec6bc874de9ac4fc4ad595cc498: Deleting WAL directory at /tmp/dist-test-task1JRg7N/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781955746769051-7083-0/minicluster-data/ts-2/wal/wals/73233d369a6b4370b74119c49e73f0f4
I20260620 11:46:17.652889 25621 ts_tablet_manager.cc:1939] T 73233d369a6b4370b74119c49e73f0f4 P f632aec6bc874de9ac4fc4ad595cc498: Deleting consensus metadata
W20260620 11:46:17.653175 25110 catalog_manager.cc:4584] TS 65d7c74f35e648e181d1eb2f8419924e (127.6.234.193:38993): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 73233d369a6b4370b74119c49e73f0f4: Network error: Client connection negotiation failed: client connection to 127.6.234.193:38993: connect: Connection refused (error 111)
I20260620 11:46:17.653426 25109 catalog_manager.cc:4841] TS f632aec6bc874de9ac4fc4ad595cc498 (127.6.234.195:41325): tablet 73233d369a6b4370b74119c49e73f0f4 (table test-table [id=a1c01c5e2b1742ecaa5dbb024d0f9557]) successfully deleted
I20260620 11:46:17.656744 7083 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task1JRg7N/build/release/bin/kudu with pid 25453
I20260620 11:46:17.661513 7083 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task1JRg7N/build/release/bin/kudu with pid 25091
2026-06-20T11:46:17Z chronyd exiting
[ OK ] EncryptionPolicyTest.TestEncryptionPolicy/2 (1940 ms)
[ RUN ] EncryptionPolicyTest.TestEncryptionPolicy/3
2026-06-20T11:46:17Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-20T11:46:17Z Disabled control of system clock
I20260620 11:46:17.688607 7083 external_mini_cluster.cc:1326] Running /tmp/dist-test-task1JRg7N/build/release/bin/kudu
/tmp/dist-test-task1JRg7N/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task1JRg7N/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781955746769051-7083-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task1JRg7N/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781955746769051-7083-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task1JRg7N/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781955746769051-7083-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task1JRg7N/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781955746769051-7083-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.6.234.254:37941
--webserver_interface=127.6.234.254
--webserver_port=0
--builtin_ntp_servers=127.6.234.212:37177
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.6.234.254:37941
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
W20260620 11:46:17.767715 25628 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 11:46:17.767938 25628 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 11:46:17.767959 25628 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 11:46:17.769042 25628 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 11:46:17.769078 25628 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 11:46:17.769093 25628 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 11:46:17.769105 25628 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 11:46:17.769117 25628 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 11:46:17.770488 25628 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.6.234.212:37177
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task1JRg7N/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781955746769051-7083-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task1JRg7N/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781955746769051-7083-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.6.234.254:37941
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.6.234.254:37941
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task1JRg7N/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781955746769051-7083-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.6.234.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.25628
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task1JRg7N/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781955746769051-7083-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 20 Jun 2026 10:32:37 UTC on ccycloud.quasar-bgyofv.root.comops.site
build id 4457173
I20260620 11:46:17.770692 25628 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 11:46:17.770921 25628 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 11:46:17.775774 25634 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 11:46:17.775782 25628 server_base.cc:1034] running on GCE node
W20260620 11:46:17.775799 25636 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 11:46:17.775786 25633 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 11:46:17.776284 25628 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 11:46:17.776592 25628 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 11:46:17.777771 25628 hybrid_clock.cc:648] HybridClock initialized: now 1781955977777744 us; error 49 us; skew 500 ppm
I20260620 11:46:17.778952 25628 webserver.cc:460] Webserver started at http://127.6.234.254:36485/ using document root <none> and password file <none>
I20260620 11:46:17.779208 25628 fs_manager.cc:359] Metadata directory not provided
I20260620 11:46:17.779277 25628 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 11:46:17.779433 25628 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 11:46:17.780496 25628 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task1JRg7N/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781955746769051-7083-0/minicluster-data/master-0/data/instance:
uuid: "9c143ae2599b418ebfacca3ca03a11c3"
format_stamp: "Formatted at 2026-06-20 11:46:17 on dist-test-slave-cjc6"
server_key: "a92b5f3d8265103efe2491fb0dcab557"
server_key_iv: "53b772965c07ad7068d5d656ad2a65ee"
server_key_version: "encryptionkey@0"
I20260620 11:46:17.780870 25628 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task1JRg7N/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781955746769051-7083-0/minicluster-data/master-0/wal/instance:
uuid: "9c143ae2599b418ebfacca3ca03a11c3"
format_stamp: "Formatted at 2026-06-20 11:46:17 on dist-test-slave-cjc6"
server_key: "a92b5f3d8265103efe2491fb0dcab557"
server_key_iv: "53b772965c07ad7068d5d656ad2a65ee"
server_key_version: "encryptionkey@0"
I20260620 11:46:17.782243 25628 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.003s
I20260620 11:46:17.783087 25642 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 11:46:17.783260 25628 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.001s
I20260620 11:46:17.783367 25628 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task1JRg7N/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781955746769051-7083-0/minicluster-data/master-0/data,/tmp/dist-test-task1JRg7N/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781955746769051-7083-0/minicluster-data/master-0/wal
uuid: "9c143ae2599b418ebfacca3ca03a11c3"
format_stamp: "Formatted at 2026-06-20 11:46:17 on dist-test-slave-cjc6"
server_key: "a92b5f3d8265103efe2491fb0dcab557"
server_key_iv: "53b772965c07ad7068d5d656ad2a65ee"
server_key_version: "encryptionkey@0"
I20260620 11:46:17.783448 25628 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task1JRg7N/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781955746769051-7083-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task1JRg7N/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781955746769051-7083-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task1JRg7N/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781955746769051-7083-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 11:46:17.796344 25628 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 11:46:17.796648 25628 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 11:46:17.796803 25628 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 11:46:17.800663 25628 rpc_server.cc:307] RPC server started. Bound to: 127.6.234.254:37941
I20260620 11:46:17.800694 25694 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.6.234.254:37941 every 8 connection(s)
I20260620 11:46:17.801031 25628 server_base.cc:1166] Dumped server information to /tmp/dist-test-task1JRg7N/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781955746769051-7083-0/minicluster-data/master-0/data/info.pb
I20260620 11:46:17.802017 25695 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 11:46:17.802671 7083 external_mini_cluster.cc:1388] Started /tmp/dist-test-task1JRg7N/build/release/bin/kudu as pid 25628
I20260620 11:46:17.802783 7083 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task1JRg7N/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781955746769051-7083-0/minicluster-data/master-0/wal/instance
I20260620 11:46:17.803006 7083 external_mini_cluster.cc:1428] Setting key 83017517a84f3a14d40ebbd127e09f7d
I20260620 11:46:17.804986 25695 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 9c143ae2599b418ebfacca3ca03a11c3: Bootstrap starting.
I20260620 11:46:17.805596 25695 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 9c143ae2599b418ebfacca3ca03a11c3: Neither blocks nor log segments found. Creating new log.
I20260620 11:46:17.805943 25695 log.cc:824] T 00000000000000000000000000000000 P 9c143ae2599b418ebfacca3ca03a11c3: Log is configured to *not* fsync() on all Append() calls
I20260620 11:46:17.806711 25695 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 9c143ae2599b418ebfacca3ca03a11c3: No bootstrap required, opened a new log
I20260620 11:46:17.808535 25695 raft_consensus.cc:348] T 00000000000000000000000000000000 P 9c143ae2599b418ebfacca3ca03a11c3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9c143ae2599b418ebfacca3ca03a11c3" member_type: VOTER last_known_addr { host: "127.6.234.254" port: 37941 } }
I20260620 11:46:17.808665 25695 raft_consensus.cc:374] T 00000000000000000000000000000000 P 9c143ae2599b418ebfacca3ca03a11c3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 11:46:17.808688 25695 raft_consensus.cc:729] T 00000000000000000000000000000000 P 9c143ae2599b418ebfacca3ca03a11c3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9c143ae2599b418ebfacca3ca03a11c3, State: Initialized, Role: FOLLOWER
I20260620 11:46:17.808702 25698 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 11:46:17.803545 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41854 (local address 127.6.234.254:37941)
0620 11:46:17.803738 (+ 193us) server_negotiation.cc:206] Beginning negotiation
0620 11:46:17.803746 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0620 11:46:17.803771 (+ 25us) server_negotiation.cc:407] Connection header received
0620 11:46:17.804224 (+ 453us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 11:46:17.804231 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 11:46:17.804290 (+ 59us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 11:46:17.804409 (+ 119us) server_negotiation.cc:226] Negotiated authn=SASL
0620 11:46:17.804641 (+ 232us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 11:46:17.805534 (+ 893us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 11:46:17.806603 (+ 1069us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 11:46:17.807036 (+ 433us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 11:46:17.807141 (+ 105us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 11:46:17.807831 (+ 690us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 11:46:17.807834 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 11:46:17.807834 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 11:46:17.807989 (+ 155us) server_negotiation.cc:922] calling sasl_server_start()
0620 11:46:17.808020 (+ 31us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 11:46:17.808030 (+ 10us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 11:46:17.808094 (+ 64us) server_negotiation.cc:1035] Waiting for connection context
0620 11:46:17.808195 (+ 101us) server_negotiation.cc:299] Negotiation successful
0620 11:46:17.808277 (+ 82us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":114,"thread_start_us":86,"threads_started":1}
I20260620 11:46:17.808853 25695 consensus_queue.cc:260] T 00000000000000000000000000000000 P 9c143ae2599b418ebfacca3ca03a11c3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9c143ae2599b418ebfacca3ca03a11c3" member_type: VOTER last_known_addr { host: "127.6.234.254" port: 37941 } }
I20260620 11:46:17.808935 25695 raft_consensus.cc:388] T 00000000000000000000000000000000 P 9c143ae2599b418ebfacca3ca03a11c3 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260620 11:46:17.808959 25695 raft_consensus.cc:482] T 00000000000000000000000000000000 P 9c143ae2599b418ebfacca3ca03a11c3 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260620 11:46:17.809252 25695 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 9c143ae2599b418ebfacca3ca03a11c3 [term 0 FOLLOWER]: Advancing to term 1
I20260620 11:46:17.809839 25695 raft_consensus.cc:504] T 00000000000000000000000000000000 P 9c143ae2599b418ebfacca3ca03a11c3 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9c143ae2599b418ebfacca3ca03a11c3" member_type: VOTER last_known_addr { host: "127.6.234.254" port: 37941 } }
I20260620 11:46:17.809953 25695 leader_election.cc:302] T 00000000000000000000000000000000 P 9c143ae2599b418ebfacca3ca03a11c3 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 9c143ae2599b418ebfacca3ca03a11c3; no voters:
I20260620 11:46:17.810135 25695 leader_election.cc:288] T 00000000000000000000000000000000 P 9c143ae2599b418ebfacca3ca03a11c3 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260620 11:46:17.810185 25700 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 9c143ae2599b418ebfacca3ca03a11c3 [term 1 FOLLOWER]: Leader election won for term 1
I20260620 11:46:17.810344 25700 raft_consensus.cc:686] T 00000000000000000000000000000000 P 9c143ae2599b418ebfacca3ca03a11c3 [term 1 LEADER]: Becoming Leader. State: Replica: 9c143ae2599b418ebfacca3ca03a11c3, State: Running, Role: LEADER
I20260620 11:46:17.810391 25695 sys_catalog.cc:564] T 00000000000000000000000000000000 P 9c143ae2599b418ebfacca3ca03a11c3 [sys.catalog]: configured and running, proceeding with master startup.
I20260620 11:46:17.810509 25700 consensus_queue.cc:237] T 00000000000000000000000000000000 P 9c143ae2599b418ebfacca3ca03a11c3 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9c143ae2599b418ebfacca3ca03a11c3" member_type: VOTER last_known_addr { host: "127.6.234.254" port: 37941 } }
I20260620 11:46:17.811683 25702 sys_catalog.cc:455] T 00000000000000000000000000000000 P 9c143ae2599b418ebfacca3ca03a11c3 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 9c143ae2599b418ebfacca3ca03a11c3. Latest consensus state: current_term: 1 leader_uuid: "9c143ae2599b418ebfacca3ca03a11c3" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9c143ae2599b418ebfacca3ca03a11c3" member_type: VOTER last_known_addr { host: "127.6.234.254" port: 37941 } } }
I20260620 11:46:17.811775 25702 sys_catalog.cc:458] T 00000000000000000000000000000000 P 9c143ae2599b418ebfacca3ca03a11c3 [sys.catalog]: This master's current role is: LEADER
I20260620 11:46:17.811882 25701 sys_catalog.cc:455] T 00000000000000000000000000000000 P 9c143ae2599b418ebfacca3ca03a11c3 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "9c143ae2599b418ebfacca3ca03a11c3" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9c143ae2599b418ebfacca3ca03a11c3" member_type: VOTER last_known_addr { host: "127.6.234.254" port: 37941 } } }
I20260620 11:46:17.812152 25701 sys_catalog.cc:458] T 00000000000000000000000000000000 P 9c143ae2599b418ebfacca3ca03a11c3 [sys.catalog]: This master's current role is: LEADER
W20260620 11:46:17.812224 25715 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 9c143ae2599b418ebfacca3ca03a11c3: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260620 11:46:17.812282 25715 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260620 11:46:17.812345 25716 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260620 11:46:17.812548 25716 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260620 11:46:17.814002 25716 catalog_manager.cc:1347] Generated new cluster ID: d5aaf4115a8745a4a5439632de4d6269
I20260620 11:46:17.814064 25716 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260620 11:46:17.818109 25716 catalog_manager.cc:1370] Generated new certificate authority record
I20260620 11:46:17.818593 25716 catalog_manager.cc:1504] Loading token signing keys...
I20260620 11:46:17.820765 25716 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 9c143ae2599b418ebfacca3ca03a11c3: Generated new TSK 0
I20260620 11:46:17.820931 25716 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260620 11:46:17.827150 7083 external_mini_cluster.cc:1326] Running /tmp/dist-test-task1JRg7N/build/release/bin/kudu
/tmp/dist-test-task1JRg7N/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task1JRg7N/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781955746769051-7083-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-task1JRg7N/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781955746769051-7083-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task1JRg7N/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781955746769051-7083-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-task1JRg7N/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781955746769051-7083-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.6.234.193:0
--local_ip_for_outbound_sockets=127.6.234.193
--webserver_interface=127.6.234.193
--webserver_port=0
--tserver_master_addrs=127.6.234.254:37941
--builtin_ntp_servers=127.6.234.212:37177
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
W20260620 11:46:17.912467 25719 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 11:46:17.912704 25719 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 11:46:17.912741 25719 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 11:46:17.913949 25719 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 11:46:17.914004 25719 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 11:46:17.914045 25719 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.6.234.193
I20260620 11:46:17.915393 25719 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.6.234.212:37177
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task1JRg7N/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781955746769051-7083-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-task1JRg7N/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781955746769051-7083-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.6.234.193:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task1JRg7N/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781955746769051-7083-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.6.234.193
--webserver_port=0
--tserver_master_addrs=127.6.234.254:37941
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.25719
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.6.234.193
--log_dir=/tmp/dist-test-task1JRg7N/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781955746769051-7083-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 20 Jun 2026 10:32:37 UTC on ccycloud.quasar-bgyofv.root.comops.site
build id 4457173
I20260620 11:46:17.915648 25719 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 11:46:17.915932 25719 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 11:46:17.921675 25725 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 11:46:17.921682 25724 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 11:46:17.922040 25719 server_base.cc:1034] running on GCE node
W20260620 11:46:17.921696 25727 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 11:46:17.922322 25719 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 11:46:17.922592 25719 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 11:46:17.923743 25719 hybrid_clock.cc:648] HybridClock initialized: now 1781955977923732 us; error 25 us; skew 500 ppm
I20260620 11:46:17.924985 25719 webserver.cc:460] Webserver started at http://127.6.234.193:35293/ using document root <none> and password file <none>
I20260620 11:46:17.925244 25719 fs_manager.cc:359] Metadata directory not provided
I20260620 11:46:17.925316 25719 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 11:46:17.925450 25719 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 11:46:17.926735 25719 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task1JRg7N/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781955746769051-7083-0/minicluster-data/ts-0/data/instance:
uuid: "4075553c1da34e6882c5980df5594aaf"
format_stamp: "Formatted at 2026-06-20 11:46:17 on dist-test-slave-cjc6"
server_key: "2b86f87ec606c1a1328a468da78e1d9f"
server_key_iv: "63036e81c288f4f72404ecb29090b3ed"
server_key_version: "encryptionkey@0"
I20260620 11:46:17.927197 25719 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task1JRg7N/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781955746769051-7083-0/minicluster-data/ts-0/wal/instance:
uuid: "4075553c1da34e6882c5980df5594aaf"
format_stamp: "Formatted at 2026-06-20 11:46:17 on dist-test-slave-cjc6"
server_key: "2b86f87ec606c1a1328a468da78e1d9f"
server_key_iv: "63036e81c288f4f72404ecb29090b3ed"
server_key_version: "encryptionkey@0"
I20260620 11:46:17.928566 25719 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.003s
I20260620 11:46:17.929332 25733 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 11:46:17.929484 25719 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.001s
I20260620 11:46:17.929579 25719 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task1JRg7N/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781955746769051-7083-0/minicluster-data/ts-0/data,/tmp/dist-test-task1JRg7N/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781955746769051-7083-0/minicluster-data/ts-0/wal
uuid: "4075553c1da34e6882c5980df5594aaf"
format_stamp: "Formatted at 2026-06-20 11:46:17 on dist-test-slave-cjc6"
server_key: "2b86f87ec606c1a1328a468da78e1d9f"
server_key_iv: "63036e81c288f4f72404ecb29090b3ed"
server_key_version: "encryptionkey@0"
I20260620 11:46:17.929649 25719 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task1JRg7N/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781955746769051-7083-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-task1JRg7N/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781955746769051-7083-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-task1JRg7N/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781955746769051-7083-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 11:46:17.945576 25719 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 11:46:17.945947 25719 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 11:46:17.946096 25719 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 11:46:17.946336 25719 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 11:46:17.946681 25719 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 11:46:17.946717 25719 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 11:46:17.946739 25719 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 11:46:17.946786 25719 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 11:46:17.952421 25719 rpc_server.cc:307] RPC server started. Bound to: 127.6.234.193:34073
I20260620 11:46:17.952489 25846 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.6.234.193:34073 every 8 connection(s)
I20260620 11:46:17.952793 25719 server_base.cc:1166] Dumped server information to /tmp/dist-test-task1JRg7N/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781955746769051-7083-0/minicluster-data/ts-0/data/info.pb
I20260620 11:46:17.957716 25698 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 11:46:17.953514 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.6.234.193:58179 (local address 127.6.234.254:37941)
0620 11:46:17.953585 (+ 71us) server_negotiation.cc:206] Beginning negotiation
0620 11:46:17.953587 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 11:46:17.953858 (+ 271us) server_negotiation.cc:407] Connection header received
0620 11:46:17.954348 (+ 490us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 11:46:17.954349 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 11:46:17.954360 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 11:46:17.954414 (+ 54us) server_negotiation.cc:226] Negotiated authn=SASL
0620 11:46:17.955080 (+ 666us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 11:46:17.955569 (+ 489us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 11:46:17.956472 (+ 903us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 11:46:17.956656 (+ 184us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 11:46:17.956704 (+ 48us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 11:46:17.957467 (+ 763us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 11:46:17.957468 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 11:46:17.957469 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 11:46:17.957473 (+ 4us) server_negotiation.cc:922] calling sasl_server_start()
0620 11:46:17.957486 (+ 13us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 11:46:17.957490 (+ 4us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 11:46:17.957539 (+ 49us) server_negotiation.cc:1035] Waiting for connection context
0620 11:46:17.957616 (+ 77us) server_negotiation.cc:299] Negotiation successful
0620 11:46:17.957643 (+ 27us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":46}
I20260620 11:46:17.957969 25849 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 11:46:17.953497 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.234.254:37941 (local address 127.6.234.193:58179)
0620 11:46:17.953740 (+ 243us) negotiation.cc:107] Waiting for socket to connect
0620 11:46:17.953789 (+ 49us) client_negotiation.cc:174] Beginning negotiation
0620 11:46:17.954259 (+ 470us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 11:46:17.954509 (+ 250us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 11:46:17.954513 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 11:46:17.954522 (+ 9us) client_negotiation.cc:189] Negotiated authn=SASL
0620 11:46:17.955008 (+ 486us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 11:46:17.955012 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 11:46:17.955643 (+ 631us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 11:46:17.955643 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 11:46:17.956400 (+ 757us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 11:46:17.956401 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 11:46:17.956726 (+ 325us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 11:46:17.956727 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 11:46:17.956831 (+ 104us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 11:46:17.957341 (+ 510us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0620 11:46:17.957341 (+ 0us) client_negotiation.cc:656] calling sasl_client_start()
0620 11:46:17.957368 (+ 27us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0620 11:46:17.957389 (+ 21us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 11:46:17.957547 (+ 158us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 11:46:17.957551 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 11:46:17.957553 (+ 2us) client_negotiation.cc:769] Sending connection context
0620 11:46:17.957605 (+ 52us) client_negotiation.cc:240] Negotiation successful
0620 11:46:17.957678 (+ 73us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":146,"thread_start_us":100,"threads_started":1}
I20260620 11:46:17.958516 25847 heartbeater.cc:347] Connected to a master server at 127.6.234.254:37941
I20260620 11:46:17.958595 25847 heartbeater.cc:464] Registering TS with master...
I20260620 11:46:17.958729 25847 heartbeater.cc:511] Master 127.6.234.254:37941 requested a full tablet report, sending...
I20260620 11:46:17.959195 25659 ts_manager.cc:194] Registered new tserver with Master: 4075553c1da34e6882c5980df5594aaf (127.6.234.193:34073)
I20260620 11:46:17.959769 25659 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.6.234.193:58179
I20260620 11:46:17.962751 7083 external_mini_cluster.cc:1388] Started /tmp/dist-test-task1JRg7N/build/release/bin/kudu as pid 25719
I20260620 11:46:17.962843 7083 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task1JRg7N/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781955746769051-7083-0/minicluster-data/ts-0/wal/instance
I20260620 11:46:17.963006 7083 external_mini_cluster.cc:1428] Setting key 01acd254ec2ceb8b18a06ca78da437b5
I20260620 11:46:17.963850 7083 external_mini_cluster.cc:1326] Running /tmp/dist-test-task1JRg7N/build/release/bin/kudu
/tmp/dist-test-task1JRg7N/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task1JRg7N/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781955746769051-7083-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-task1JRg7N/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781955746769051-7083-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task1JRg7N/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781955746769051-7083-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-task1JRg7N/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781955746769051-7083-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.6.234.194:0
--local_ip_for_outbound_sockets=127.6.234.194
--webserver_interface=127.6.234.194
--webserver_port=0
--tserver_master_addrs=127.6.234.254:37941
--builtin_ntp_servers=127.6.234.212:37177
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
W20260620 11:46:18.046432 25850 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 11:46:18.046648 25850 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 11:46:18.046669 25850 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 11:46:18.047937 25850 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 11:46:18.047979 25850 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 11:46:18.048012 25850 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.6.234.194
I20260620 11:46:18.049330 25850 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.6.234.212:37177
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task1JRg7N/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781955746769051-7083-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-task1JRg7N/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781955746769051-7083-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.6.234.194:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task1JRg7N/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781955746769051-7083-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.6.234.194
--webserver_port=0
--tserver_master_addrs=127.6.234.254:37941
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.25850
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.6.234.194
--log_dir=/tmp/dist-test-task1JRg7N/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781955746769051-7083-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 20 Jun 2026 10:32:37 UTC on ccycloud.quasar-bgyofv.root.comops.site
build id 4457173
I20260620 11:46:18.049571 25850 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 11:46:18.049916 25850 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 11:46:18.054855 25855 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 11:46:18.054877 25858 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 11:46:18.054865 25856 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 11:46:18.055088 25850 server_base.cc:1034] running on GCE node
I20260620 11:46:18.055291 25850 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 11:46:18.055536 25850 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 11:46:18.056699 25850 hybrid_clock.cc:648] HybridClock initialized: now 1781955978056689 us; error 39 us; skew 500 ppm
I20260620 11:46:18.058063 25850 webserver.cc:460] Webserver started at http://127.6.234.194:38215/ using document root <none> and password file <none>
I20260620 11:46:18.058310 25850 fs_manager.cc:359] Metadata directory not provided
I20260620 11:46:18.058379 25850 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 11:46:18.058522 25850 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 11:46:18.059552 25850 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task1JRg7N/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781955746769051-7083-0/minicluster-data/ts-1/data/instance:
uuid: "03fef4f4908043a3b9ce242fe21d7b56"
format_stamp: "Formatted at 2026-06-20 11:46:18 on dist-test-slave-cjc6"
server_key: "5e8e77f4f0ff76377ab1fa400189563a"
server_key_iv: "8ed2e590019852d9fb0455de71e6a0fc"
server_key_version: "encryptionkey@0"
I20260620 11:46:18.059911 25850 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task1JRg7N/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781955746769051-7083-0/minicluster-data/ts-1/wal/instance:
uuid: "03fef4f4908043a3b9ce242fe21d7b56"
format_stamp: "Formatted at 2026-06-20 11:46:18 on dist-test-slave-cjc6"
server_key: "5e8e77f4f0ff76377ab1fa400189563a"
server_key_iv: "8ed2e590019852d9fb0455de71e6a0fc"
server_key_version: "encryptionkey@0"
I20260620 11:46:18.061192 25850 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260620 11:46:18.061946 25864 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 11:46:18.062125 25850 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260620 11:46:18.062222 25850 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task1JRg7N/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781955746769051-7083-0/minicluster-data/ts-1/data,/tmp/dist-test-task1JRg7N/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781955746769051-7083-0/minicluster-data/ts-1/wal
uuid: "03fef4f4908043a3b9ce242fe21d7b56"
format_stamp: "Formatted at 2026-06-20 11:46:18 on dist-test-slave-cjc6"
server_key: "5e8e77f4f0ff76377ab1fa400189563a"
server_key_iv: "8ed2e590019852d9fb0455de71e6a0fc"
server_key_version: "encryptionkey@0"
I20260620 11:46:18.062301 25850 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task1JRg7N/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781955746769051-7083-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-task1JRg7N/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781955746769051-7083-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-task1JRg7N/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781955746769051-7083-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 11:46:18.080915 25850 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 11:46:18.081218 25850 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 11:46:18.081372 25850 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 11:46:18.081606 25850 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 11:46:18.081959 25850 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 11:46:18.082013 25850 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 11:46:18.082063 25850 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 11:46:18.082093 25850 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 11:46:18.088013 25850 rpc_server.cc:307] RPC server started. Bound to: 127.6.234.194:36951
I20260620 11:46:18.088065 25977 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.6.234.194:36951 every 8 connection(s)
I20260620 11:46:18.088378 25850 server_base.cc:1166] Dumped server information to /tmp/dist-test-task1JRg7N/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781955746769051-7083-0/minicluster-data/ts-1/data/info.pb
I20260620 11:46:18.088682 7083 external_mini_cluster.cc:1388] Started /tmp/dist-test-task1JRg7N/build/release/bin/kudu as pid 25850
I20260620 11:46:18.088800 7083 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task1JRg7N/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781955746769051-7083-0/minicluster-data/ts-1/wal/instance
I20260620 11:46:18.089058 7083 external_mini_cluster.cc:1428] Setting key 74a45ddedad55c1d509bd06a2ba37c10
I20260620 11:46:18.090090 7083 external_mini_cluster.cc:1326] Running /tmp/dist-test-task1JRg7N/build/release/bin/kudu
/tmp/dist-test-task1JRg7N/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task1JRg7N/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781955746769051-7083-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-task1JRg7N/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781955746769051-7083-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task1JRg7N/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781955746769051-7083-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-task1JRg7N/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781955746769051-7083-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.6.234.195:0
--local_ip_for_outbound_sockets=127.6.234.195
--webserver_interface=127.6.234.195
--webserver_port=0
--tserver_master_addrs=127.6.234.254:37941
--builtin_ntp_servers=127.6.234.212:37177
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
I20260620 11:46:18.094303 25698 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 11:46:18.089097 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.6.234.194:41627 (local address 127.6.234.254:37941)
0620 11:46:18.089151 (+ 54us) server_negotiation.cc:206] Beginning negotiation
0620 11:46:18.089153 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 11:46:18.089441 (+ 288us) server_negotiation.cc:407] Connection header received
0620 11:46:18.090168 (+ 727us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 11:46:18.090168 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 11:46:18.090185 (+ 17us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 11:46:18.090255 (+ 70us) server_negotiation.cc:226] Negotiated authn=SASL
0620 11:46:18.090981 (+ 726us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 11:46:18.091630 (+ 649us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 11:46:18.092827 (+ 1197us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 11:46:18.093037 (+ 210us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 11:46:18.093124 (+ 87us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 11:46:18.093997 (+ 873us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 11:46:18.093998 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 11:46:18.093998 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 11:46:18.094003 (+ 5us) server_negotiation.cc:922] calling sasl_server_start()
0620 11:46:18.094018 (+ 15us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 11:46:18.094022 (+ 4us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 11:46:18.094092 (+ 70us) server_negotiation.cc:1035] Waiting for connection context
0620 11:46:18.094194 (+ 102us) server_negotiation.cc:299] Negotiation successful
0620 11:46:18.094227 (+ 33us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":27}
I20260620 11:46:18.094511 25980 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 11:46:18.089079 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.234.254:37941 (local address 127.6.234.194:41627)
0620 11:46:18.089273 (+ 194us) negotiation.cc:107] Waiting for socket to connect
0620 11:46:18.089314 (+ 41us) client_negotiation.cc:174] Beginning negotiation
0620 11:46:18.090048 (+ 734us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 11:46:18.090379 (+ 331us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 11:46:18.090383 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 11:46:18.090391 (+ 8us) client_negotiation.cc:189] Negotiated authn=SASL
0620 11:46:18.090879 (+ 488us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 11:46:18.090883 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 11:46:18.091721 (+ 838us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 11:46:18.091722 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 11:46:18.092722 (+ 1000us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 11:46:18.092724 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 11:46:18.093132 (+ 408us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 11:46:18.093133 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 11:46:18.093263 (+ 130us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 11:46:18.093834 (+ 571us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0620 11:46:18.093834 (+ 0us) client_negotiation.cc:656] calling sasl_client_start()
0620 11:46:18.093863 (+ 29us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0620 11:46:18.093885 (+ 22us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 11:46:18.094091 (+ 206us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 11:46:18.094098 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 11:46:18.094101 (+ 3us) client_negotiation.cc:769] Sending connection context
0620 11:46:18.094185 (+ 84us) client_negotiation.cc:240] Negotiation successful
0620 11:46:18.094283 (+ 98us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":120,"thread_start_us":82,"threads_started":1}
I20260620 11:46:18.094956 25978 heartbeater.cc:347] Connected to a master server at 127.6.234.254:37941
I20260620 11:46:18.095073 25978 heartbeater.cc:464] Registering TS with master...
I20260620 11:46:18.095283 25978 heartbeater.cc:511] Master 127.6.234.254:37941 requested a full tablet report, sending...
I20260620 11:46:18.095679 25659 ts_manager.cc:194] Registered new tserver with Master: 03fef4f4908043a3b9ce242fe21d7b56 (127.6.234.194:36951)
I20260620 11:46:18.096235 25659 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.6.234.194:41627
W20260620 11:46:18.171509 25981 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 11:46:18.171702 25981 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 11:46:18.171723 25981 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 11:46:18.172834 25981 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 11:46:18.172868 25981 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 11:46:18.172892 25981 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.6.234.195
I20260620 11:46:18.174211 25981 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.6.234.212:37177
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task1JRg7N/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781955746769051-7083-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-task1JRg7N/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781955746769051-7083-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.6.234.195:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task1JRg7N/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781955746769051-7083-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.6.234.195
--webserver_port=0
--tserver_master_addrs=127.6.234.254:37941
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.25981
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.6.234.195
--log_dir=/tmp/dist-test-task1JRg7N/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781955746769051-7083-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 20 Jun 2026 10:32:37 UTC on ccycloud.quasar-bgyofv.root.comops.site
build id 4457173
I20260620 11:46:18.174429 25981 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 11:46:18.174676 25981 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 11:46:18.179514 25986 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 11:46:18.179533 25981 server_base.cc:1034] running on GCE node
W20260620 11:46:18.179586 25987 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 11:46:18.179512 25989 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 11:46:18.179937 25981 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 11:46:18.180135 25981 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 11:46:18.181267 25981 hybrid_clock.cc:648] HybridClock initialized: now 1781955978181257 us; error 27 us; skew 500 ppm
I20260620 11:46:18.182535 25981 webserver.cc:460] Webserver started at http://127.6.234.195:34775/ using document root <none> and password file <none>
I20260620 11:46:18.182785 25981 fs_manager.cc:359] Metadata directory not provided
I20260620 11:46:18.182830 25981 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 11:46:18.183010 25981 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 11:46:18.184043 25981 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task1JRg7N/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781955746769051-7083-0/minicluster-data/ts-2/data/instance:
uuid: "bd61728cddd64eb6a47e36efb4cd67f3"
format_stamp: "Formatted at 2026-06-20 11:46:18 on dist-test-slave-cjc6"
server_key: "b028442dad52caa97ec9245dfed7cd05"
server_key_iv: "df160bb69715fb24c2f45e1bc68f3cfb"
server_key_version: "encryptionkey@0"
I20260620 11:46:18.184415 25981 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task1JRg7N/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781955746769051-7083-0/minicluster-data/ts-2/wal/instance:
uuid: "bd61728cddd64eb6a47e36efb4cd67f3"
format_stamp: "Formatted at 2026-06-20 11:46:18 on dist-test-slave-cjc6"
server_key: "b028442dad52caa97ec9245dfed7cd05"
server_key_iv: "df160bb69715fb24c2f45e1bc68f3cfb"
server_key_version: "encryptionkey@0"
I20260620 11:46:18.185730 25981 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260620 11:46:18.186587 25995 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 11:46:18.186906 25981 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260620 11:46:18.187014 25981 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task1JRg7N/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781955746769051-7083-0/minicluster-data/ts-2/data,/tmp/dist-test-task1JRg7N/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781955746769051-7083-0/minicluster-data/ts-2/wal
uuid: "bd61728cddd64eb6a47e36efb4cd67f3"
format_stamp: "Formatted at 2026-06-20 11:46:18 on dist-test-slave-cjc6"
server_key: "b028442dad52caa97ec9245dfed7cd05"
server_key_iv: "df160bb69715fb24c2f45e1bc68f3cfb"
server_key_version: "encryptionkey@0"
I20260620 11:46:18.187098 25981 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task1JRg7N/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781955746769051-7083-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-task1JRg7N/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781955746769051-7083-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-task1JRg7N/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781955746769051-7083-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 11:46:18.197522 25981 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 11:46:18.197842 25981 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 11:46:18.197999 25981 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 11:46:18.198216 25981 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 11:46:18.198545 25981 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 11:46:18.198601 25981 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 11:46:18.198637 25981 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 11:46:18.198664 25981 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 11:46:18.204510 25981 rpc_server.cc:307] RPC server started. Bound to: 127.6.234.195:46467
I20260620 11:46:18.204563 26108 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.6.234.195:46467 every 8 connection(s)
I20260620 11:46:18.204897 25981 server_base.cc:1166] Dumped server information to /tmp/dist-test-task1JRg7N/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781955746769051-7083-0/minicluster-data/ts-2/data/info.pb
I20260620 11:46:18.209900 25698 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 11:46:18.205641 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.6.234.195:41333 (local address 127.6.234.254:37941)
0620 11:46:18.205726 (+ 85us) server_negotiation.cc:206] Beginning negotiation
0620 11:46:18.205728 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 11:46:18.205915 (+ 187us) server_negotiation.cc:407] Connection header received
0620 11:46:18.206374 (+ 459us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 11:46:18.206375 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 11:46:18.206384 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 11:46:18.206441 (+ 57us) server_negotiation.cc:226] Negotiated authn=SASL
0620 11:46:18.207110 (+ 669us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 11:46:18.207606 (+ 496us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 11:46:18.208566 (+ 960us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 11:46:18.208750 (+ 184us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 11:46:18.208816 (+ 66us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 11:46:18.209574 (+ 758us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 11:46:18.209576 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 11:46:18.209576 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 11:46:18.209580 (+ 4us) server_negotiation.cc:922] calling sasl_server_start()
0620 11:46:18.209595 (+ 15us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 11:46:18.209599 (+ 4us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 11:46:18.209646 (+ 47us) server_negotiation.cc:1035] Waiting for connection context
0620 11:46:18.209776 (+ 130us) server_negotiation.cc:299] Negotiation successful
0620 11:46:18.209803 (+ 27us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":20}
I20260620 11:46:18.210067 26111 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 11:46:18.205589 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.234.254:37941 (local address 127.6.234.195:41333)
0620 11:46:18.205787 (+ 198us) negotiation.cc:107] Waiting for socket to connect
0620 11:46:18.205827 (+ 40us) client_negotiation.cc:174] Beginning negotiation
0620 11:46:18.206293 (+ 466us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 11:46:18.206496 (+ 203us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 11:46:18.206499 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 11:46:18.206507 (+ 8us) client_negotiation.cc:189] Negotiated authn=SASL
0620 11:46:18.206995 (+ 488us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 11:46:18.206999 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 11:46:18.207692 (+ 693us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 11:46:18.207692 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 11:46:18.208475 (+ 783us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 11:46:18.208477 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 11:46:18.208833 (+ 356us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 11:46:18.208834 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 11:46:18.208938 (+ 104us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 11:46:18.209434 (+ 496us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0620 11:46:18.209434 (+ 0us) client_negotiation.cc:656] calling sasl_client_start()
0620 11:46:18.209461 (+ 27us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0620 11:46:18.209484 (+ 23us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 11:46:18.209656 (+ 172us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 11:46:18.209660 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 11:46:18.209662 (+ 2us) client_negotiation.cc:769] Sending connection context
0620 11:46:18.209736 (+ 74us) client_negotiation.cc:240] Negotiation successful
0620 11:46:18.209867 (+ 131us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":136,"thread_start_us":87,"threads_started":1}
I20260620 11:46:18.210575 26109 heartbeater.cc:347] Connected to a master server at 127.6.234.254:37941
I20260620 11:46:18.210654 26109 heartbeater.cc:464] Registering TS with master...
I20260620 11:46:18.210891 26109 heartbeater.cc:511] Master 127.6.234.254:37941 requested a full tablet report, sending...
I20260620 11:46:18.211294 25659 ts_manager.cc:194] Registered new tserver with Master: bd61728cddd64eb6a47e36efb4cd67f3 (127.6.234.195:46467)
I20260620 11:46:18.211676 25659 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.6.234.195:41333
I20260620 11:46:18.214955 7083 external_mini_cluster.cc:1388] Started /tmp/dist-test-task1JRg7N/build/release/bin/kudu as pid 25981
I20260620 11:46:18.215073 7083 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task1JRg7N/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781955746769051-7083-0/minicluster-data/ts-2/wal/instance
I20260620 11:46:18.215260 7083 external_mini_cluster.cc:1428] Setting key 9a026e078778e08354e30e77d4fde72f
I20260620 11:46:18.216174 7083 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260620 11:46:18.220250 25698 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 11:46:18.217575 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41870 (local address 127.6.234.254:37941)
0620 11:46:18.217631 (+ 56us) server_negotiation.cc:206] Beginning negotiation
0620 11:46:18.217633 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 11:46:18.217674 (+ 41us) server_negotiation.cc:407] Connection header received
0620 11:46:18.217783 (+ 109us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 11:46:18.217786 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 11:46:18.217798 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 11:46:18.217847 (+ 49us) server_negotiation.cc:226] Negotiated authn=SASL
0620 11:46:18.217995 (+ 148us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 11:46:18.218445 (+ 450us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 11:46:18.219130 (+ 685us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 11:46:18.219300 (+ 170us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 11:46:18.219348 (+ 48us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 11:46:18.220026 (+ 678us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 11:46:18.220028 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 11:46:18.220028 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 11:46:18.220032 (+ 4us) server_negotiation.cc:922] calling sasl_server_start()
0620 11:46:18.220046 (+ 14us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 11:46:18.220050 (+ 4us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 11:46:18.220130 (+ 80us) server_negotiation.cc:1035] Waiting for connection context
0620 11:46:18.220189 (+ 59us) server_negotiation.cc:299] Negotiation successful
0620 11:46:18.220199 (+ 10us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":35}
I20260620 11:46:18.221735 25659 catalog_manager.cc:1908] Servicing CreateTable request from {username='slave'} at 127.0.0.1:41870:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260620 11:46:18.223320 25659 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260620 11:46:18.229498 26117 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 11:46:18.225554 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.234.195:46467 (local address 127.0.0.1:53334)
0620 11:46:18.225739 (+ 185us) negotiation.cc:107] Waiting for socket to connect
0620 11:46:18.225783 (+ 44us) client_negotiation.cc:174] Beginning negotiation
0620 11:46:18.225848 (+ 65us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 11:46:18.226269 (+ 421us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 11:46:18.226272 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 11:46:18.226275 (+ 3us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 11:46:18.226340 (+ 65us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 11:46:18.226342 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 11:46:18.227346 (+ 1004us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 11:46:18.227347 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 11:46:18.228173 (+ 826us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 11:46:18.228175 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 11:46:18.229190 (+ 1015us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 11:46:18.229191 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 11:46:18.229272 (+ 81us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 11:46:18.229285 (+ 13us) client_negotiation.cc:769] Sending connection context
0620 11:46:18.229361 (+ 76us) client_negotiation.cc:240] Negotiation successful
0620 11:46:18.229413 (+ 52us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":127,"thread_start_us":76,"threads_started":1}
I20260620 11:46:18.229513 26118 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 11:46:18.225655 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53334 (local address 127.6.234.195:46467)
0620 11:46:18.225842 (+ 187us) server_negotiation.cc:206] Beginning negotiation
0620 11:46:18.225848 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0620 11:46:18.225860 (+ 12us) server_negotiation.cc:407] Connection header received
0620 11:46:18.225954 (+ 94us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 11:46:18.225957 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 11:46:18.225974 (+ 17us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 11:46:18.226063 (+ 89us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 11:46:18.226421 (+ 358us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 11:46:18.227261 (+ 840us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 11:46:18.228268 (+ 1007us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 11:46:18.228986 (+ 718us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 11:46:18.229047 (+ 61us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 11:46:18.229157 (+ 110us) server_negotiation.cc:1035] Waiting for connection context
0620 11:46:18.229388 (+ 231us) server_negotiation.cc:299] Negotiation successful
0620 11:46:18.229430 (+ 42us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":148,"thread_start_us":114,"threads_started":1}
I20260620 11:46:18.230010 26122 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 11:46:18.226178 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55396 (local address 127.6.234.193:34073)
0620 11:46:18.226357 (+ 179us) server_negotiation.cc:206] Beginning negotiation
0620 11:46:18.226361 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 11:46:18.226370 (+ 9us) server_negotiation.cc:407] Connection header received
0620 11:46:18.226407 (+ 37us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 11:46:18.226409 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 11:46:18.226425 (+ 16us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 11:46:18.226528 (+ 103us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 11:46:18.226643 (+ 115us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 11:46:18.227744 (+ 1101us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 11:46:18.229032 (+ 1288us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 11:46:18.229685 (+ 653us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 11:46:18.229793 (+ 108us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 11:46:18.229861 (+ 68us) server_negotiation.cc:1035] Waiting for connection context
0620 11:46:18.229921 (+ 60us) server_negotiation.cc:299] Negotiation successful
0620 11:46:18.229952 (+ 31us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":146,"thread_start_us":72,"threads_started":1}
I20260620 11:46:18.230024 26119 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 11:46:18.225804 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.234.193:34073 (local address 127.0.0.1:55396)
0620 11:46:18.226161 (+ 357us) negotiation.cc:107] Waiting for socket to connect
0620 11:46:18.226171 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0620 11:46:18.226219 (+ 48us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 11:46:18.226525 (+ 306us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 11:46:18.226526 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 11:46:18.226528 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 11:46:18.226561 (+ 33us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 11:46:18.226562 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 11:46:18.227834 (+ 1272us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 11:46:18.227835 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 11:46:18.228930 (+ 1095us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 11:46:18.228933 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 11:46:18.229791 (+ 858us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 11:46:18.229792 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 11:46:18.229852 (+ 60us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 11:46:18.229855 (+ 3us) client_negotiation.cc:769] Sending connection context
0620 11:46:18.229912 (+ 57us) client_negotiation.cc:240] Negotiation successful
0620 11:46:18.229953 (+ 41us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":338,"thread_start_us":80,"threads_started":1}
I20260620 11:46:18.230557 26043 tablet_service.cc:1467] Processing CreateTablet for tablet 833773ad806b449095e7c993e9a68665 (DEFAULT_TABLE table=test-table [id=c544ee9a4b0e4a6d840e90f0ae55d2c2]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 11:46:18.230712 25781 tablet_service.cc:1467] Processing CreateTablet for tablet 833773ad806b449095e7c993e9a68665 (DEFAULT_TABLE table=test-table [id=c544ee9a4b0e4a6d840e90f0ae55d2c2]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 11:46:18.231031 26043 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 833773ad806b449095e7c993e9a68665. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 11:46:18.231042 25781 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 833773ad806b449095e7c993e9a68665. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 11:46:18.234439 26123 tablet_bootstrap.cc:492] T 833773ad806b449095e7c993e9a68665 P bd61728cddd64eb6a47e36efb4cd67f3: Bootstrap starting.
I20260620 11:46:18.235095 26123 tablet_bootstrap.cc:654] T 833773ad806b449095e7c993e9a68665 P bd61728cddd64eb6a47e36efb4cd67f3: Neither blocks nor log segments found. Creating new log.
I20260620 11:46:18.235167 26120 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 11:46:18.225980 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.234.194:36951 (local address 127.0.0.1:43982)
0620 11:46:18.226185 (+ 205us) negotiation.cc:107] Waiting for socket to connect
0620 11:46:18.226194 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0620 11:46:18.226232 (+ 38us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 11:46:18.226919 (+ 687us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 11:46:18.226920 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 11:46:18.226922 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 11:46:18.226954 (+ 32us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 11:46:18.226956 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 11:46:18.228164 (+ 1208us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 11:46:18.228165 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 11:46:18.229055 (+ 890us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 11:46:18.229058 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 11:46:18.234892 (+ 5834us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 11:46:18.234893 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 11:46:18.234966 (+ 73us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 11:46:18.234971 (+ 5us) client_negotiation.cc:769] Sending connection context
0620 11:46:18.235039 (+ 68us) client_negotiation.cc:240] Negotiation successful
0620 11:46:18.235089 (+ 50us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":188,"thread_start_us":62,"threads_started":1}
I20260620 11:46:18.235188 26121 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 11:46:18.226092 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43982 (local address 127.6.234.194:36951)
0620 11:46:18.226659 (+ 567us) server_negotiation.cc:206] Beginning negotiation
0620 11:46:18.226664 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 11:46:18.226674 (+ 10us) server_negotiation.cc:407] Connection header received
0620 11:46:18.226726 (+ 52us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 11:46:18.226727 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 11:46:18.226750 (+ 23us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 11:46:18.226852 (+ 102us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 11:46:18.227027 (+ 175us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 11:46:18.228082 (+ 1055us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 11:46:18.229137 (+ 1055us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 11:46:18.234780 (+ 5643us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 11:46:18.234841 (+ 61us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 11:46:18.234898 (+ 57us) server_negotiation.cc:1035] Waiting for connection context
0620 11:46:18.235057 (+ 159us) server_negotiation.cc:299] Negotiation successful
0620 11:46:18.235092 (+ 35us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":524,"thread_start_us":93,"threads_started":1}
I20260620 11:46:18.235401 26123 log.cc:824] T 833773ad806b449095e7c993e9a68665 P bd61728cddd64eb6a47e36efb4cd67f3: Log is configured to *not* fsync() on all Append() calls
I20260620 11:46:18.235775 26124 tablet_bootstrap.cc:492] T 833773ad806b449095e7c993e9a68665 P 4075553c1da34e6882c5980df5594aaf: Bootstrap starting.
I20260620 11:46:18.235966 25912 tablet_service.cc:1467] Processing CreateTablet for tablet 833773ad806b449095e7c993e9a68665 (DEFAULT_TABLE table=test-table [id=c544ee9a4b0e4a6d840e90f0ae55d2c2]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 11:46:18.236127 26123 tablet_bootstrap.cc:492] T 833773ad806b449095e7c993e9a68665 P bd61728cddd64eb6a47e36efb4cd67f3: No bootstrap required, opened a new log
I20260620 11:46:18.236173 26123 ts_tablet_manager.cc:1397] T 833773ad806b449095e7c993e9a68665 P bd61728cddd64eb6a47e36efb4cd67f3: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260620 11:46:18.236315 25912 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 833773ad806b449095e7c993e9a68665. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 11:46:18.236398 26124 tablet_bootstrap.cc:654] T 833773ad806b449095e7c993e9a68665 P 4075553c1da34e6882c5980df5594aaf: Neither blocks nor log segments found. Creating new log.
I20260620 11:46:18.236689 26124 log.cc:824] T 833773ad806b449095e7c993e9a68665 P 4075553c1da34e6882c5980df5594aaf: Log is configured to *not* fsync() on all Append() calls
I20260620 11:46:18.237514 26124 tablet_bootstrap.cc:492] T 833773ad806b449095e7c993e9a68665 P 4075553c1da34e6882c5980df5594aaf: No bootstrap required, opened a new log
I20260620 11:46:18.237576 26124 ts_tablet_manager.cc:1397] T 833773ad806b449095e7c993e9a68665 P 4075553c1da34e6882c5980df5594aaf: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260620 11:46:18.238322 26123 raft_consensus.cc:348] T 833773ad806b449095e7c993e9a68665 P bd61728cddd64eb6a47e36efb4cd67f3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bd61728cddd64eb6a47e36efb4cd67f3" member_type: VOTER last_known_addr { host: "127.6.234.195" port: 46467 } } peers { permanent_uuid: "4075553c1da34e6882c5980df5594aaf" member_type: VOTER last_known_addr { host: "127.6.234.193" port: 34073 } } peers { permanent_uuid: "03fef4f4908043a3b9ce242fe21d7b56" member_type: VOTER last_known_addr { host: "127.6.234.194" port: 36951 } }
I20260620 11:46:18.238472 26123 raft_consensus.cc:374] T 833773ad806b449095e7c993e9a68665 P bd61728cddd64eb6a47e36efb4cd67f3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 11:46:18.238515 26123 raft_consensus.cc:729] T 833773ad806b449095e7c993e9a68665 P bd61728cddd64eb6a47e36efb4cd67f3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: bd61728cddd64eb6a47e36efb4cd67f3, State: Initialized, Role: FOLLOWER
I20260620 11:46:18.238633 26123 consensus_queue.cc:260] T 833773ad806b449095e7c993e9a68665 P bd61728cddd64eb6a47e36efb4cd67f3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bd61728cddd64eb6a47e36efb4cd67f3" member_type: VOTER last_known_addr { host: "127.6.234.195" port: 46467 } } peers { permanent_uuid: "4075553c1da34e6882c5980df5594aaf" member_type: VOTER last_known_addr { host: "127.6.234.193" port: 34073 } } peers { permanent_uuid: "03fef4f4908043a3b9ce242fe21d7b56" member_type: VOTER last_known_addr { host: "127.6.234.194" port: 36951 } }
I20260620 11:46:18.238874 26123 ts_tablet_manager.cc:1428] T 833773ad806b449095e7c993e9a68665 P bd61728cddd64eb6a47e36efb4cd67f3: Time spent starting tablet: real 0.003s user 0.000s sys 0.003s
I20260620 11:46:18.239006 26109 heartbeater.cc:503] Master 127.6.234.254:37941 was elected leader, sending a full tablet report...
I20260620 11:46:18.240269 26124 raft_consensus.cc:348] T 833773ad806b449095e7c993e9a68665 P 4075553c1da34e6882c5980df5594aaf [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bd61728cddd64eb6a47e36efb4cd67f3" member_type: VOTER last_known_addr { host: "127.6.234.195" port: 46467 } } peers { permanent_uuid: "4075553c1da34e6882c5980df5594aaf" member_type: VOTER last_known_addr { host: "127.6.234.193" port: 34073 } } peers { permanent_uuid: "03fef4f4908043a3b9ce242fe21d7b56" member_type: VOTER last_known_addr { host: "127.6.234.194" port: 36951 } }
I20260620 11:46:18.240401 26124 raft_consensus.cc:374] T 833773ad806b449095e7c993e9a68665 P 4075553c1da34e6882c5980df5594aaf [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 11:46:18.240423 26124 raft_consensus.cc:729] T 833773ad806b449095e7c993e9a68665 P 4075553c1da34e6882c5980df5594aaf [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4075553c1da34e6882c5980df5594aaf, State: Initialized, Role: FOLLOWER
I20260620 11:46:18.240562 26124 consensus_queue.cc:260] T 833773ad806b449095e7c993e9a68665 P 4075553c1da34e6882c5980df5594aaf [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bd61728cddd64eb6a47e36efb4cd67f3" member_type: VOTER last_known_addr { host: "127.6.234.195" port: 46467 } } peers { permanent_uuid: "4075553c1da34e6882c5980df5594aaf" member_type: VOTER last_known_addr { host: "127.6.234.193" port: 34073 } } peers { permanent_uuid: "03fef4f4908043a3b9ce242fe21d7b56" member_type: VOTER last_known_addr { host: "127.6.234.194" port: 36951 } }
I20260620 11:46:18.240800 26124 ts_tablet_manager.cc:1428] T 833773ad806b449095e7c993e9a68665 P 4075553c1da34e6882c5980df5594aaf: Time spent starting tablet: real 0.003s user 0.003s sys 0.000s
I20260620 11:46:18.240886 25847 heartbeater.cc:503] Master 127.6.234.254:37941 was elected leader, sending a full tablet report...
I20260620 11:46:18.241065 26128 tablet_bootstrap.cc:492] T 833773ad806b449095e7c993e9a68665 P 03fef4f4908043a3b9ce242fe21d7b56: Bootstrap starting.
I20260620 11:46:18.241762 26128 tablet_bootstrap.cc:654] T 833773ad806b449095e7c993e9a68665 P 03fef4f4908043a3b9ce242fe21d7b56: Neither blocks nor log segments found. Creating new log.
I20260620 11:46:18.242079 26128 log.cc:824] T 833773ad806b449095e7c993e9a68665 P 03fef4f4908043a3b9ce242fe21d7b56: Log is configured to *not* fsync() on all Append() calls
I20260620 11:46:18.242712 26128 tablet_bootstrap.cc:492] T 833773ad806b449095e7c993e9a68665 P 03fef4f4908043a3b9ce242fe21d7b56: No bootstrap required, opened a new log
I20260620 11:46:18.242764 26128 ts_tablet_manager.cc:1397] T 833773ad806b449095e7c993e9a68665 P 03fef4f4908043a3b9ce242fe21d7b56: Time spent bootstrapping tablet: real 0.002s user 0.000s sys 0.001s
I20260620 11:46:18.244444 26128 raft_consensus.cc:348] T 833773ad806b449095e7c993e9a68665 P 03fef4f4908043a3b9ce242fe21d7b56 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bd61728cddd64eb6a47e36efb4cd67f3" member_type: VOTER last_known_addr { host: "127.6.234.195" port: 46467 } } peers { permanent_uuid: "4075553c1da34e6882c5980df5594aaf" member_type: VOTER last_known_addr { host: "127.6.234.193" port: 34073 } } peers { permanent_uuid: "03fef4f4908043a3b9ce242fe21d7b56" member_type: VOTER last_known_addr { host: "127.6.234.194" port: 36951 } }
I20260620 11:46:18.244575 26128 raft_consensus.cc:374] T 833773ad806b449095e7c993e9a68665 P 03fef4f4908043a3b9ce242fe21d7b56 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 11:46:18.244597 26128 raft_consensus.cc:729] T 833773ad806b449095e7c993e9a68665 P 03fef4f4908043a3b9ce242fe21d7b56 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 03fef4f4908043a3b9ce242fe21d7b56, State: Initialized, Role: FOLLOWER
I20260620 11:46:18.244711 26128 consensus_queue.cc:260] T 833773ad806b449095e7c993e9a68665 P 03fef4f4908043a3b9ce242fe21d7b56 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bd61728cddd64eb6a47e36efb4cd67f3" member_type: VOTER last_known_addr { host: "127.6.234.195" port: 46467 } } peers { permanent_uuid: "4075553c1da34e6882c5980df5594aaf" member_type: VOTER last_known_addr { host: "127.6.234.193" port: 34073 } } peers { permanent_uuid: "03fef4f4908043a3b9ce242fe21d7b56" member_type: VOTER last_known_addr { host: "127.6.234.194" port: 36951 } }
I20260620 11:46:18.244931 26128 ts_tablet_manager.cc:1428] T 833773ad806b449095e7c993e9a68665 P 03fef4f4908043a3b9ce242fe21d7b56: Time spent starting tablet: real 0.002s user 0.000s sys 0.002s
I20260620 11:46:18.245021 25978 heartbeater.cc:503] Master 127.6.234.254:37941 was elected leader, sending a full tablet report...
I20260620 11:46:18.328903 26129 raft_consensus.cc:482] T 833773ad806b449095e7c993e9a68665 P 4075553c1da34e6882c5980df5594aaf [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260620 11:46:18.329057 26129 raft_consensus.cc:504] T 833773ad806b449095e7c993e9a68665 P 4075553c1da34e6882c5980df5594aaf [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bd61728cddd64eb6a47e36efb4cd67f3" member_type: VOTER last_known_addr { host: "127.6.234.195" port: 46467 } } peers { permanent_uuid: "4075553c1da34e6882c5980df5594aaf" member_type: VOTER last_known_addr { host: "127.6.234.193" port: 34073 } } peers { permanent_uuid: "03fef4f4908043a3b9ce242fe21d7b56" member_type: VOTER last_known_addr { host: "127.6.234.194" port: 36951 } }
I20260620 11:46:18.329423 26129 leader_election.cc:288] T 833773ad806b449095e7c993e9a68665 P 4075553c1da34e6882c5980df5594aaf [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers bd61728cddd64eb6a47e36efb4cd67f3 (127.6.234.195:46467), 03fef4f4908043a3b9ce242fe21d7b56 (127.6.234.194:36951)
I20260620 11:46:18.332374 25849 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 11:46:18.329628 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.234.195:46467 (local address 127.6.234.193:58109)
0620 11:46:18.329683 (+ 55us) negotiation.cc:107] Waiting for socket to connect
0620 11:46:18.329722 (+ 39us) client_negotiation.cc:174] Beginning negotiation
0620 11:46:18.329770 (+ 48us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 11:46:18.330028 (+ 258us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 11:46:18.330029 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 11:46:18.330031 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 11:46:18.330109 (+ 78us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 11:46:18.330111 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 11:46:18.330786 (+ 675us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 11:46:18.330788 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 11:46:18.331576 (+ 788us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 11:46:18.331579 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 11:46:18.332121 (+ 542us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 11:46:18.332122 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 11:46:18.332193 (+ 71us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 11:46:18.332197 (+ 4us) client_negotiation.cc:769] Sending connection context
0620 11:46:18.332255 (+ 58us) client_negotiation.cc:240] Negotiation successful
0620 11:46:18.332301 (+ 46us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":31}
I20260620 11:46:18.332404 26118 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 11:46:18.329751 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.6.234.193:58109 (local address 127.6.234.195:46467)
0620 11:46:18.329820 (+ 69us) server_negotiation.cc:206] Beginning negotiation
0620 11:46:18.329822 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 11:46:18.329833 (+ 11us) server_negotiation.cc:407] Connection header received
0620 11:46:18.329867 (+ 34us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 11:46:18.329868 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 11:46:18.329878 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 11:46:18.329990 (+ 112us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 11:46:18.330183 (+ 193us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 11:46:18.330715 (+ 532us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 11:46:18.331663 (+ 948us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 11:46:18.332058 (+ 395us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 11:46:18.332111 (+ 53us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 11:46:18.332118 (+ 7us) server_negotiation.cc:1035] Waiting for connection context
0620 11:46:18.332299 (+ 181us) server_negotiation.cc:299] Negotiation successful
0620 11:46:18.332333 (+ 34us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":45}
I20260620 11:46:18.332625 26121 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 11:46:18.329860 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.6.234.193:36515 (local address 127.6.234.194:36951)
0620 11:46:18.329925 (+ 65us) server_negotiation.cc:206] Beginning negotiation
0620 11:46:18.329927 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 11:46:18.330035 (+ 108us) server_negotiation.cc:407] Connection header received
0620 11:46:18.330089 (+ 54us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 11:46:18.330090 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 11:46:18.330099 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 11:46:18.330162 (+ 63us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 11:46:18.330315 (+ 153us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 11:46:18.330790 (+ 475us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 11:46:18.331724 (+ 934us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 11:46:18.332232 (+ 508us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 11:46:18.332296 (+ 64us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 11:46:18.332303 (+ 7us) server_negotiation.cc:1035] Waiting for connection context
0620 11:46:18.332489 (+ 186us) server_negotiation.cc:299] Negotiation successful
0620 11:46:18.332564 (+ 75us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":38}
I20260620 11:46:18.332763 26063 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "833773ad806b449095e7c993e9a68665" candidate_uuid: "4075553c1da34e6882c5980df5594aaf" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "bd61728cddd64eb6a47e36efb4cd67f3" is_pre_election: true
I20260620 11:46:18.332863 26132 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 11:46:18.329789 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.234.194:36951 (local address 127.6.234.193:36515)
0620 11:46:18.329959 (+ 170us) negotiation.cc:107] Waiting for socket to connect
0620 11:46:18.329969 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0620 11:46:18.330028 (+ 59us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 11:46:18.330183 (+ 155us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 11:46:18.330183 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 11:46:18.330185 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 11:46:18.330249 (+ 64us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 11:46:18.330251 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 11:46:18.330868 (+ 617us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 11:46:18.330869 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 11:46:18.331634 (+ 765us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 11:46:18.331636 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 11:46:18.332310 (+ 674us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 11:46:18.332311 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 11:46:18.332391 (+ 80us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 11:46:18.332394 (+ 3us) client_negotiation.cc:769] Sending connection context
0620 11:46:18.332744 (+ 350us) client_negotiation.cc:240] Negotiation successful
0620 11:46:18.332791 (+ 47us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":148,"mutex_wait_us":13,"thread_start_us":113,"threads_started":1}
I20260620 11:46:18.332918 26063 raft_consensus.cc:2454] T 833773ad806b449095e7c993e9a68665 P bd61728cddd64eb6a47e36efb4cd67f3 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4075553c1da34e6882c5980df5594aaf in term 0.
I20260620 11:46:18.333171 25735 leader_election.cc:302] T 833773ad806b449095e7c993e9a68665 P 4075553c1da34e6882c5980df5594aaf [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4075553c1da34e6882c5980df5594aaf, bd61728cddd64eb6a47e36efb4cd67f3; no voters:
I20260620 11:46:18.333261 25932 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "833773ad806b449095e7c993e9a68665" candidate_uuid: "4075553c1da34e6882c5980df5594aaf" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "03fef4f4908043a3b9ce242fe21d7b56" is_pre_election: true
I20260620 11:46:18.333297 26129 raft_consensus.cc:2781] T 833773ad806b449095e7c993e9a68665 P 4075553c1da34e6882c5980df5594aaf [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260620 11:46:18.333344 26129 raft_consensus.cc:482] T 833773ad806b449095e7c993e9a68665 P 4075553c1da34e6882c5980df5594aaf [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260620 11:46:18.333365 26129 raft_consensus.cc:3037] T 833773ad806b449095e7c993e9a68665 P 4075553c1da34e6882c5980df5594aaf [term 0 FOLLOWER]: Advancing to term 1
I20260620 11:46:18.333403 25932 raft_consensus.cc:2454] T 833773ad806b449095e7c993e9a68665 P 03fef4f4908043a3b9ce242fe21d7b56 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4075553c1da34e6882c5980df5594aaf in term 0.
I20260620 11:46:18.334189 26129 raft_consensus.cc:504] T 833773ad806b449095e7c993e9a68665 P 4075553c1da34e6882c5980df5594aaf [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bd61728cddd64eb6a47e36efb4cd67f3" member_type: VOTER last_known_addr { host: "127.6.234.195" port: 46467 } } peers { permanent_uuid: "4075553c1da34e6882c5980df5594aaf" member_type: VOTER last_known_addr { host: "127.6.234.193" port: 34073 } } peers { permanent_uuid: "03fef4f4908043a3b9ce242fe21d7b56" member_type: VOTER last_known_addr { host: "127.6.234.194" port: 36951 } }
I20260620 11:46:18.334321 26129 leader_election.cc:288] T 833773ad806b449095e7c993e9a68665 P 4075553c1da34e6882c5980df5594aaf [CANDIDATE]: Term 1 election: Requested vote from peers bd61728cddd64eb6a47e36efb4cd67f3 (127.6.234.195:46467), 03fef4f4908043a3b9ce242fe21d7b56 (127.6.234.194:36951)
I20260620 11:46:18.334532 25932 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "833773ad806b449095e7c993e9a68665" candidate_uuid: "4075553c1da34e6882c5980df5594aaf" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "03fef4f4908043a3b9ce242fe21d7b56"
I20260620 11:46:18.334527 26063 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "833773ad806b449095e7c993e9a68665" candidate_uuid: "4075553c1da34e6882c5980df5594aaf" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "bd61728cddd64eb6a47e36efb4cd67f3"
I20260620 11:46:18.334607 26063 raft_consensus.cc:3037] T 833773ad806b449095e7c993e9a68665 P bd61728cddd64eb6a47e36efb4cd67f3 [term 0 FOLLOWER]: Advancing to term 1
I20260620 11:46:18.334607 25932 raft_consensus.cc:3037] T 833773ad806b449095e7c993e9a68665 P 03fef4f4908043a3b9ce242fe21d7b56 [term 0 FOLLOWER]: Advancing to term 1
I20260620 11:46:18.335480 25932 raft_consensus.cc:2454] T 833773ad806b449095e7c993e9a68665 P 03fef4f4908043a3b9ce242fe21d7b56 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4075553c1da34e6882c5980df5594aaf in term 1.
I20260620 11:46:18.335480 26063 raft_consensus.cc:2454] T 833773ad806b449095e7c993e9a68665 P bd61728cddd64eb6a47e36efb4cd67f3 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4075553c1da34e6882c5980df5594aaf in term 1.
I20260620 11:46:18.335651 25735 leader_election.cc:302] T 833773ad806b449095e7c993e9a68665 P 4075553c1da34e6882c5980df5594aaf [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 03fef4f4908043a3b9ce242fe21d7b56, 4075553c1da34e6882c5980df5594aaf; no voters:
I20260620 11:46:18.335747 26129 raft_consensus.cc:2781] T 833773ad806b449095e7c993e9a68665 P 4075553c1da34e6882c5980df5594aaf [term 1 FOLLOWER]: Leader election won for term 1
I20260620 11:46:18.335889 26129 raft_consensus.cc:686] T 833773ad806b449095e7c993e9a68665 P 4075553c1da34e6882c5980df5594aaf [term 1 LEADER]: Becoming Leader. State: Replica: 4075553c1da34e6882c5980df5594aaf, State: Running, Role: LEADER
I20260620 11:46:18.335985 26129 consensus_queue.cc:237] T 833773ad806b449095e7c993e9a68665 P 4075553c1da34e6882c5980df5594aaf [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bd61728cddd64eb6a47e36efb4cd67f3" member_type: VOTER last_known_addr { host: "127.6.234.195" port: 46467 } } peers { permanent_uuid: "4075553c1da34e6882c5980df5594aaf" member_type: VOTER last_known_addr { host: "127.6.234.193" port: 34073 } } peers { permanent_uuid: "03fef4f4908043a3b9ce242fe21d7b56" member_type: VOTER last_known_addr { host: "127.6.234.194" port: 36951 } }
I20260620 11:46:18.336808 25658 catalog_manager.cc:5495] T 833773ad806b449095e7c993e9a68665 P 4075553c1da34e6882c5980df5594aaf reported cstate change: term changed from 0 to 1, leader changed from <none> to 4075553c1da34e6882c5980df5594aaf (127.6.234.193). New cstate: current_term: 1 leader_uuid: "4075553c1da34e6882c5980df5594aaf" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bd61728cddd64eb6a47e36efb4cd67f3" member_type: VOTER last_known_addr { host: "127.6.234.195" port: 46467 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "4075553c1da34e6882c5980df5594aaf" member_type: VOTER last_known_addr { host: "127.6.234.193" port: 34073 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "03fef4f4908043a3b9ce242fe21d7b56" member_type: VOTER last_known_addr { host: "127.6.234.194" port: 36951 } health_report { overall_health: UNKNOWN } } }
W20260620 11:46:18.338950 25979 tablet.cc:2369] T 833773ad806b449095e7c993e9a68665 P 03fef4f4908043a3b9ce242fe21d7b56: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260620 11:46:18.345590 26122 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 11:46:18.343284 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55408 (local address 127.6.234.193:34073)
0620 11:46:18.343337 (+ 53us) server_negotiation.cc:206] Beginning negotiation
0620 11:46:18.343339 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 11:46:18.343348 (+ 9us) server_negotiation.cc:407] Connection header received
0620 11:46:18.343385 (+ 37us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 11:46:18.343386 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 11:46:18.343402 (+ 16us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 11:46:18.343481 (+ 79us) server_negotiation.cc:226] Negotiated authn=TOKEN
0620 11:46:18.343609 (+ 128us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 11:46:18.344069 (+ 460us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 11:46:18.344832 (+ 763us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 11:46:18.345026 (+ 194us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 11:46:18.345104 (+ 78us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 11:46:18.345254 (+ 150us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0620 11:46:18.345344 (+ 90us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0620 11:46:18.345460 (+ 116us) server_negotiation.cc:1035] Waiting for connection context
0620 11:46:18.345489 (+ 29us) server_negotiation.cc:299] Negotiation successful
0620 11:46:18.345500 (+ 11us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":34}
I20260620 11:46:18.347234 26063 raft_consensus.cc:1261] T 833773ad806b449095e7c993e9a68665 P bd61728cddd64eb6a47e36efb4cd67f3 [term 1 FOLLOWER]: Refusing update from remote peer 4075553c1da34e6882c5980df5594aaf: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 11:46:18.347245 25932 raft_consensus.cc:1261] T 833773ad806b449095e7c993e9a68665 P 03fef4f4908043a3b9ce242fe21d7b56 [term 1 FOLLOWER]: Refusing update from remote peer 4075553c1da34e6882c5980df5594aaf: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 11:46:18.347590 26133 consensus_queue.cc:1035] T 833773ad806b449095e7c993e9a68665 P 4075553c1da34e6882c5980df5594aaf [LEADER]: Connected to new peer: Peer: permanent_uuid: "bd61728cddd64eb6a47e36efb4cd67f3" member_type: VOTER last_known_addr { host: "127.6.234.195" port: 46467 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 11:46:18.347723 26129 consensus_queue.cc:1035] T 833773ad806b449095e7c993e9a68665 P 4075553c1da34e6882c5980df5594aaf [LEADER]: Connected to new peer: Peer: permanent_uuid: "03fef4f4908043a3b9ce242fe21d7b56" member_type: VOTER last_known_addr { host: "127.6.234.194" port: 36951 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 11:46:18.350232 26138 mvcc.cc:205] Tried to move back new op lower bound from 7298891687307829248 to 7298891687264886784. Current Snapshot: MvccSnapshot[applied={T|T < 7298891687307829248}]
I20260620 11:46:18.352252 25658 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:41870:
table { table_name: "test-table" } modify_external_catalogs: true
I20260620 11:46:18.352342 25658 catalog_manager.cc:2645] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:41870:
table { table_name: "test-table" } modify_external_catalogs: true
I20260620 11:46:18.353082 25658 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 9c143ae2599b418ebfacca3ca03a11c3: Sending DeleteTablet for 3 replicas of tablet 833773ad806b449095e7c993e9a68665
I20260620 11:46:18.353405 26043 tablet_service.cc:1514] Processing DeleteTablet for tablet 833773ad806b449095e7c993e9a68665 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 11:46:18 UTC) from {username='slave'} at 127.0.0.1:53334
I20260620 11:46:18.353399 25781 tablet_service.cc:1514] Processing DeleteTablet for tablet 833773ad806b449095e7c993e9a68665 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 11:46:18 UTC) from {username='slave'} at 127.0.0.1:55396
I20260620 11:46:18.353688 25912 tablet_service.cc:1514] Processing DeleteTablet for tablet 833773ad806b449095e7c993e9a68665 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 11:46:18 UTC) from {username='slave'} at 127.0.0.1:43982
I20260620 11:46:18.353798 26146 tablet_replica.cc:330] T 833773ad806b449095e7c993e9a68665 P bd61728cddd64eb6a47e36efb4cd67f3: stopping tablet replica
I20260620 11:46:18.353893 26146 raft_consensus.cc:2229] T 833773ad806b449095e7c993e9a68665 P bd61728cddd64eb6a47e36efb4cd67f3 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260620 11:46:18.354004 26146 raft_consensus.cc:2258] T 833773ad806b449095e7c993e9a68665 P bd61728cddd64eb6a47e36efb4cd67f3 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 11:46:18.354050 7083 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task1JRg7N/build/release/bin/kudu with pid 25719
I20260620 11:46:18.355062 26146 ts_tablet_manager.cc:1905] T 833773ad806b449095e7c993e9a68665 P bd61728cddd64eb6a47e36efb4cd67f3: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 11:46:18.355233 26148 tablet_replica.cc:330] T 833773ad806b449095e7c993e9a68665 P 03fef4f4908043a3b9ce242fe21d7b56: stopping tablet replica
I20260620 11:46:18.355337 26148 raft_consensus.cc:2229] T 833773ad806b449095e7c993e9a68665 P 03fef4f4908043a3b9ce242fe21d7b56 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260620 11:46:18.355435 26148 raft_consensus.cc:2258] T 833773ad806b449095e7c993e9a68665 P 03fef4f4908043a3b9ce242fe21d7b56 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 11:46:18.355844 26148 ts_tablet_manager.cc:1905] T 833773ad806b449095e7c993e9a68665 P 03fef4f4908043a3b9ce242fe21d7b56: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 11:46:18.356812 26146 ts_tablet_manager.cc:1918] T 833773ad806b449095e7c993e9a68665 P bd61728cddd64eb6a47e36efb4cd67f3: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260620 11:46:18.356916 26146 log.cc:1196] T 833773ad806b449095e7c993e9a68665 P bd61728cddd64eb6a47e36efb4cd67f3: Deleting WAL directory at /tmp/dist-test-task1JRg7N/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781955746769051-7083-0/minicluster-data/ts-2/wal/wals/833773ad806b449095e7c993e9a68665
I20260620 11:46:18.357174 26146 ts_tablet_manager.cc:1939] T 833773ad806b449095e7c993e9a68665 P bd61728cddd64eb6a47e36efb4cd67f3: Deleting consensus metadata
I20260620 11:46:18.357559 25644 catalog_manager.cc:4841] TS bd61728cddd64eb6a47e36efb4cd67f3 (127.6.234.195:46467): tablet 833773ad806b449095e7c993e9a68665 (table test-table [id=c544ee9a4b0e4a6d840e90f0ae55d2c2]) successfully deleted
I20260620 11:46:18.357807 26148 ts_tablet_manager.cc:1918] T 833773ad806b449095e7c993e9a68665 P 03fef4f4908043a3b9ce242fe21d7b56: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260620 11:46:18.357879 26148 log.cc:1196] T 833773ad806b449095e7c993e9a68665 P 03fef4f4908043a3b9ce242fe21d7b56: Deleting WAL directory at /tmp/dist-test-task1JRg7N/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781955746769051-7083-0/minicluster-data/ts-1/wal/wals/833773ad806b449095e7c993e9a68665
I20260620 11:46:18.358119 26148 ts_tablet_manager.cc:1939] T 833773ad806b449095e7c993e9a68665 P 03fef4f4908043a3b9ce242fe21d7b56: Deleting consensus metadata
I20260620 11:46:18.358503 25644 catalog_manager.cc:4841] TS 03fef4f4908043a3b9ce242fe21d7b56 (127.6.234.194:36951): tablet 833773ad806b449095e7c993e9a68665 (table test-table [id=c544ee9a4b0e4a6d840e90f0ae55d2c2]) successfully deleted
W20260620 11:46:18.360526 25644 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.6.234.193:34073 (error 108)
I20260620 11:46:18.360711 7083 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task1JRg7N/build/release/bin/kudu with pid 25850
I20260620 11:46:18.361591 26120 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 11:46:18.360894 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.234.193:34073 (local address 127.0.0.1:55422)
0620 11:46:18.361502 (+ 608us) negotiation.cc:107] Waiting for socket to connect
0620 11:46:18.361550 (+ 48us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.234.193:34073: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":582}
W20260620 11:46:18.361749 25644 catalog_manager.cc:4584] TS 4075553c1da34e6882c5980df5594aaf (127.6.234.193:34073): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 833773ad806b449095e7c993e9a68665: Network error: Client connection negotiation failed: client connection to 127.6.234.193:34073: connect: Connection refused (error 111)
I20260620 11:46:18.366077 7083 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task1JRg7N/build/release/bin/kudu with pid 25981
I20260620 11:46:18.370374 7083 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task1JRg7N/build/release/bin/kudu with pid 25628
2026-06-20T11:46:18Z chronyd exiting
[ OK ] EncryptionPolicyTest.TestEncryptionPolicy/3 (707 ms)
[ RUN ] EncryptionPolicyTest.TestEncryptionPolicy/4
Loading random data
Initializing database '/tmp/dist-test-task1JRg7N/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781955746769051-7083-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 20 11:46:18 dist-test-slave-cjc6 krb5kdc[26152](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 20 11:46:18 dist-test-slave-cjc6 krb5kdc[26152](info): set up 2 sockets
Jun 20 11:46:18 dist-test-slave-cjc6 krb5kdc[26152](info): commencing operation
krb5kdc: starting...
W20260620 11:46:19.413571 7083 mini_kdc.cc:121] Time spent starting KDC: real 1.024s user 0.000s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task1JRg7N/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781955746769051-7083-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task1JRg7N/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781955746769051-7083-0/krb5kdc/test-user.keytab.
Jun 20 11:46:19 dist-test-slave-cjc6 krb5kdc[26152](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781955979, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-20T11:46:19Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-20T11:46:19Z Disabled control of system clock
WARNING: no policy specified for kudu/127.6.234.254@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.6.234.254@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.6.234.254 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task1JRg7N/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781955746769051-7083-0/krb5kdc/kudu_127.6.234.254.keytab.
Entry for principal kudu/127.6.234.254 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task1JRg7N/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781955746769051-7083-0/krb5kdc/kudu_127.6.234.254.keytab.
I20260620 11:46:19.530501 7083 external_mini_cluster.cc:1326] Running /tmp/dist-test-task1JRg7N/build/release/bin/kudu
/tmp/dist-test-task1JRg7N/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task1JRg7N/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781955746769051-7083-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task1JRg7N/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781955746769051-7083-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task1JRg7N/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781955746769051-7083-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task1JRg7N/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781955746769051-7083-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.6.234.254:38673
--webserver_interface=127.6.234.254
--webserver_port=0
--keytab_file=/tmp/dist-test-task1JRg7N/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781955746769051-7083-0/krb5kdc/kudu_127.6.234.254.keytab
--principal=kudu/127.6.234.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.6.234.212:41711
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.6.234.254:38673
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task1JRg7N/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781955746769051-7083-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task1JRg7N/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781955746769051-7083-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task1JRg7N/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781955746769051-7083-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 11:46:19.611781 26165 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 11:46:19.612013 26165 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 11:46:19.612051 26165 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 11:46:19.613170 26165 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 11:46:19.613235 26165 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 11:46:19.613260 26165 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 11:46:19.613313 26165 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 11:46:19.613343 26165 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 11:46:19.614946 26165 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.6.234.212:41711
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task1JRg7N/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781955746769051-7083-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task1JRg7N/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781955746769051-7083-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.6.234.254:38673
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task1JRg7N/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781955746769051-7083-0/krb5kdc/kudu_127.6.234.254.keytab
--principal=kudu/127.6.234.254
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.6.234.254:38673
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task1JRg7N/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781955746769051-7083-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.6.234.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.26165
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task1JRg7N/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781955746769051-7083-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 20 Jun 2026 10:32:37 UTC on ccycloud.quasar-bgyofv.root.comops.site
build id 4457173
I20260620 11:46:19.615238 26165 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 11:46:19.615468 26165 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 11:46:19.620400 26171 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 11:46:19.620406 26170 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 11:46:19.620427 26173 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 11:46:19.620469 26165 server_base.cc:1034] running on GCE node
I20260620 11:46:19.620963 26165 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 11:46:19.621246 26165 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 11:46:19.622406 26165 hybrid_clock.cc:648] HybridClock initialized: now 1781955979622403 us; error 48 us; skew 500 ppm
Jun 20 11:46:19 dist-test-slave-cjc6 krb5kdc[26152](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781955979, etypes {rep=17 tkt=17 ses=17}, kudu/127.6.234.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 11:46:19.624771 26165 init.cc:377] Logged in from keytab as kudu/127.6.234.254@KRBTEST.COM (short username kudu)
I20260620 11:46:19.625201 26165 webserver.cc:460] Webserver started at http://127.6.234.254:44011/ using document root <none> and password file <none>
I20260620 11:46:19.625418 26165 fs_manager.cc:359] Metadata directory not provided
I20260620 11:46:19.625486 26165 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 11:46:19.625634 26165 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 11:46:19.626658 26165 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task1JRg7N/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781955746769051-7083-0/minicluster-data/master-0/data/instance:
uuid: "6150208d5e044cc9b5e38215bc21010c"
format_stamp: "Formatted at 2026-06-20 11:46:19 on dist-test-slave-cjc6"
server_key: "8fe9713b8621a49e3a4cc23e807cdd34"
server_key_iv: "79f49a001ee0dab44824c1a74dd18eb7"
server_key_version: "encryptionkey@0"
I20260620 11:46:19.627019 26165 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task1JRg7N/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781955746769051-7083-0/minicluster-data/master-0/wal/instance:
uuid: "6150208d5e044cc9b5e38215bc21010c"
format_stamp: "Formatted at 2026-06-20 11:46:19 on dist-test-slave-cjc6"
server_key: "8fe9713b8621a49e3a4cc23e807cdd34"
server_key_iv: "79f49a001ee0dab44824c1a74dd18eb7"
server_key_version: "encryptionkey@0"
I20260620 11:46:19.628296 26165 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260620 11:46:19.629029 26180 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 11:46:19.629212 26165 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260620 11:46:19.629308 26165 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task1JRg7N/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781955746769051-7083-0/minicluster-data/master-0/data,/tmp/dist-test-task1JRg7N/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781955746769051-7083-0/minicluster-data/master-0/wal
uuid: "6150208d5e044cc9b5e38215bc21010c"
format_stamp: "Formatted at 2026-06-20 11:46:19 on dist-test-slave-cjc6"
server_key: "8fe9713b8621a49e3a4cc23e807cdd34"
server_key_iv: "79f49a001ee0dab44824c1a74dd18eb7"
server_key_version: "encryptionkey@0"
I20260620 11:46:19.629391 26165 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task1JRg7N/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781955746769051-7083-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task1JRg7N/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781955746769051-7083-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task1JRg7N/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781955746769051-7083-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 11:46:19.653079 26165 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 11:46:19.655572 26165 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 11:46:19.655735 26165 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 11:46:19.660398 26165 rpc_server.cc:307] RPC server started. Bound to: 127.6.234.254:38673
I20260620 11:46:19.660427 26232 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.6.234.254:38673 every 8 connection(s)
I20260620 11:46:19.660771 26165 server_base.cc:1166] Dumped server information to /tmp/dist-test-task1JRg7N/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781955746769051-7083-0/minicluster-data/master-0/data/info.pb
I20260620 11:46:19.661638 26233 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 11:46:19.663856 26233 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 6150208d5e044cc9b5e38215bc21010c: Bootstrap starting.
I20260620 11:46:19.664430 26233 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 6150208d5e044cc9b5e38215bc21010c: Neither blocks nor log segments found. Creating new log.
I20260620 11:46:19.664716 26233 log.cc:824] T 00000000000000000000000000000000 P 6150208d5e044cc9b5e38215bc21010c: Log is configured to *not* fsync() on all Append() calls
I20260620 11:46:19.665019 7083 external_mini_cluster.cc:1388] Started /tmp/dist-test-task1JRg7N/build/release/bin/kudu as pid 26165
I20260620 11:46:19.665155 7083 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task1JRg7N/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781955746769051-7083-0/minicluster-data/master-0/wal/instance
I20260620 11:46:19.665346 7083 external_mini_cluster.cc:1428] Setting key a5c35b11ac0b8eb41066e814aa56f71e
I20260620 11:46:19.665551 26233 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 6150208d5e044cc9b5e38215bc21010c: No bootstrap required, opened a new log
I20260620 11:46:19.668709 26233 raft_consensus.cc:348] T 00000000000000000000000000000000 P 6150208d5e044cc9b5e38215bc21010c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6150208d5e044cc9b5e38215bc21010c" member_type: VOTER last_known_addr { host: "127.6.234.254" port: 38673 } }
I20260620 11:46:19.668905 26233 raft_consensus.cc:374] T 00000000000000000000000000000000 P 6150208d5e044cc9b5e38215bc21010c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 11:46:19.668948 26233 raft_consensus.cc:729] T 00000000000000000000000000000000 P 6150208d5e044cc9b5e38215bc21010c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6150208d5e044cc9b5e38215bc21010c, State: Initialized, Role: FOLLOWER
I20260620 11:46:19.669102 26233 consensus_queue.cc:260] T 00000000000000000000000000000000 P 6150208d5e044cc9b5e38215bc21010c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6150208d5e044cc9b5e38215bc21010c" member_type: VOTER last_known_addr { host: "127.6.234.254" port: 38673 } }
I20260620 11:46:19.669183 26233 raft_consensus.cc:388] T 00000000000000000000000000000000 P 6150208d5e044cc9b5e38215bc21010c [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260620 11:46:19.669216 26233 raft_consensus.cc:482] T 00000000000000000000000000000000 P 6150208d5e044cc9b5e38215bc21010c [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260620 11:46:19.669263 26233 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 6150208d5e044cc9b5e38215bc21010c [term 0 FOLLOWER]: Advancing to term 1
I20260620 11:46:19.670040 26233 raft_consensus.cc:504] T 00000000000000000000000000000000 P 6150208d5e044cc9b5e38215bc21010c [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6150208d5e044cc9b5e38215bc21010c" member_type: VOTER last_known_addr { host: "127.6.234.254" port: 38673 } }
I20260620 11:46:19.670157 26233 leader_election.cc:302] T 00000000000000000000000000000000 P 6150208d5e044cc9b5e38215bc21010c [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 6150208d5e044cc9b5e38215bc21010c; no voters:
I20260620 11:46:19.670323 26233 leader_election.cc:288] T 00000000000000000000000000000000 P 6150208d5e044cc9b5e38215bc21010c [CANDIDATE]: Term 1 election: Requested vote from peers
I20260620 11:46:19.670408 26238 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 6150208d5e044cc9b5e38215bc21010c [term 1 FOLLOWER]: Leader election won for term 1
I20260620 11:46:19.670586 26233 sys_catalog.cc:564] T 00000000000000000000000000000000 P 6150208d5e044cc9b5e38215bc21010c [sys.catalog]: configured and running, proceeding with master startup.
I20260620 11:46:19.670601 26238 raft_consensus.cc:686] T 00000000000000000000000000000000 P 6150208d5e044cc9b5e38215bc21010c [term 1 LEADER]: Becoming Leader. State: Replica: 6150208d5e044cc9b5e38215bc21010c, State: Running, Role: LEADER
I20260620 11:46:19.672029 26238 consensus_queue.cc:237] T 00000000000000000000000000000000 P 6150208d5e044cc9b5e38215bc21010c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6150208d5e044cc9b5e38215bc21010c" member_type: VOTER last_known_addr { host: "127.6.234.254" port: 38673 } }
I20260620 11:46:19.672626 26240 sys_catalog.cc:455] T 00000000000000000000000000000000 P 6150208d5e044cc9b5e38215bc21010c [sys.catalog]: SysCatalogTable state changed. Reason: New leader 6150208d5e044cc9b5e38215bc21010c. Latest consensus state: current_term: 1 leader_uuid: "6150208d5e044cc9b5e38215bc21010c" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6150208d5e044cc9b5e38215bc21010c" member_type: VOTER last_known_addr { host: "127.6.234.254" port: 38673 } } }
I20260620 11:46:19.672626 26239 sys_catalog.cc:455] T 00000000000000000000000000000000 P 6150208d5e044cc9b5e38215bc21010c [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "6150208d5e044cc9b5e38215bc21010c" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6150208d5e044cc9b5e38215bc21010c" member_type: VOTER last_known_addr { host: "127.6.234.254" port: 38673 } } }
I20260620 11:46:19.672740 26240 sys_catalog.cc:458] T 00000000000000000000000000000000 P 6150208d5e044cc9b5e38215bc21010c [sys.catalog]: This master's current role is: LEADER
I20260620 11:46:19.672746 26239 sys_catalog.cc:458] T 00000000000000000000000000000000 P 6150208d5e044cc9b5e38215bc21010c [sys.catalog]: This master's current role is: LEADER
I20260620 11:46:19.673116 26252 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260620 11:46:19.673623 26252 catalog_manager.cc:1484] Initializing Kudu cluster ID...
Jun 20 11:46:19 dist-test-slave-cjc6 krb5kdc[26152](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781955979, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.6.234.254@KRBTEST.COM
I20260620 11:46:19.675588 26252 catalog_manager.cc:1347] Generated new cluster ID: 1f2df503759549459936da29397a546c
I20260620 11:46:19.675645 26252 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260620 11:46:19.678277 26237 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 11:46:19.665889 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41532 (local address 127.6.234.254:38673)
0620 11:46:19.666058 (+ 169us) server_negotiation.cc:206] Beginning negotiation
0620 11:46:19.666067 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0620 11:46:19.666098 (+ 31us) server_negotiation.cc:407] Connection header received
0620 11:46:19.667464 (+ 1366us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 11:46:19.667487 (+ 23us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 11:46:19.667561 (+ 74us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 11:46:19.667701 (+ 140us) server_negotiation.cc:226] Negotiated authn=SASL
0620 11:46:19.668322 (+ 621us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 11:46:19.669743 (+ 1421us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 11:46:19.670751 (+ 1008us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 11:46:19.671203 (+ 452us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 11:46:19.671317 (+ 114us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 11:46:19.674403 (+ 3086us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 11:46:19.674407 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 11:46:19.674407 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 11:46:19.674419 (+ 12us) server_negotiation.cc:922] calling sasl_server_start()
0620 11:46:19.676421 (+ 2002us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 11:46:19.676814 (+ 393us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 11:46:19.676817 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 11:46:19.676818 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 11:46:19.676922 (+ 104us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 11:46:19.677107 (+ 185us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 11:46:19.677107 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 11:46:19.677108 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 11:46:19.677368 (+ 260us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 11:46:19.677480 (+ 112us) server_negotiation.cc:1035] Waiting for connection context
0620 11:46:19.677947 (+ 467us) server_negotiation.cc:299] Negotiation successful
0620 11:46:19.678082 (+ 135us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":123,"thread_start_us":95,"threads_started":1}
I20260620 11:46:19.691995 26252 catalog_manager.cc:1370] Generated new certificate authority record
I20260620 11:46:19.692505 26252 catalog_manager.cc:1504] Loading token signing keys...
I20260620 11:46:19.695659 26252 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 6150208d5e044cc9b5e38215bc21010c: Generated new TSK 0
I20260620 11:46:19.695832 26252 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.6.234.193@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.6.234.193@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.6.234.193 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task1JRg7N/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781955746769051-7083-0/krb5kdc/kudu_127.6.234.193.keytab.
Entry for principal kudu/127.6.234.193 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task1JRg7N/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781955746769051-7083-0/krb5kdc/kudu_127.6.234.193.keytab.
I20260620 11:46:19.727841 7083 external_mini_cluster.cc:1326] Running /tmp/dist-test-task1JRg7N/build/release/bin/kudu
/tmp/dist-test-task1JRg7N/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task1JRg7N/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781955746769051-7083-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-task1JRg7N/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781955746769051-7083-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task1JRg7N/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781955746769051-7083-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-task1JRg7N/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781955746769051-7083-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.6.234.193:0
--local_ip_for_outbound_sockets=127.6.234.193
--webserver_interface=127.6.234.193
--webserver_port=0
--tserver_master_addrs=127.6.234.254:38673
--keytab_file=/tmp/dist-test-task1JRg7N/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781955746769051-7083-0/krb5kdc/kudu_127.6.234.193.keytab
--principal=kudu/127.6.234.193
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.6.234.212:41711
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task1JRg7N/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781955746769051-7083-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task1JRg7N/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781955746769051-7083-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task1JRg7N/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781955746769051-7083-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 11:46:19.808578 26259 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 11:46:19.808804 26259 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 11:46:19.808843 26259 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 11:46:19.810050 26259 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 11:46:19.810114 26259 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 11:46:19.810159 26259 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.6.234.193
I20260620 11:46:19.811635 26259 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.6.234.212:41711
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task1JRg7N/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781955746769051-7083-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-task1JRg7N/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781955746769051-7083-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task1JRg7N/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781955746769051-7083-0/krb5kdc/kudu_127.6.234.193.keytab
--principal=kudu/127.6.234.193
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.6.234.193:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task1JRg7N/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781955746769051-7083-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.6.234.193
--webserver_port=0
--tserver_master_addrs=127.6.234.254:38673
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.26259
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.6.234.193
--log_dir=/tmp/dist-test-task1JRg7N/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781955746769051-7083-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 20 Jun 2026 10:32:37 UTC on ccycloud.quasar-bgyofv.root.comops.site
build id 4457173
I20260620 11:46:19.811898 26259 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 11:46:19.812153 26259 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 11:46:19.817863 26267 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 11:46:19.817871 26265 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 11:46:19.818128 26259 server_base.cc:1034] running on GCE node
W20260620 11:46:19.817873 26264 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 11:46:19.818374 26259 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 11:46:19.818643 26259 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 11:46:19.819793 26259 hybrid_clock.cc:648] HybridClock initialized: now 1781955979819776 us; error 33 us; skew 500 ppm
Jun 20 11:46:19 dist-test-slave-cjc6 krb5kdc[26152](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781955979, etypes {rep=17 tkt=17 ses=17}, kudu/127.6.234.193@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 11:46:19.822404 26259 init.cc:377] Logged in from keytab as kudu/127.6.234.193@KRBTEST.COM (short username kudu)
I20260620 11:46:19.822896 26259 webserver.cc:460] Webserver started at http://127.6.234.193:42927/ using document root <none> and password file <none>
I20260620 11:46:19.823155 26259 fs_manager.cc:359] Metadata directory not provided
I20260620 11:46:19.823202 26259 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 11:46:19.823370 26259 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 11:46:19.824380 26259 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task1JRg7N/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781955746769051-7083-0/minicluster-data/ts-0/data/instance:
uuid: "180bb34b014448ffa2daa5f7698fa698"
format_stamp: "Formatted at 2026-06-20 11:46:19 on dist-test-slave-cjc6"
server_key: "913852a14e11691304d0e0210e78bf23"
server_key_iv: "2906dcb1283cc2d1db408b98be06840b"
server_key_version: "encryptionkey@0"
I20260620 11:46:19.824730 26259 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task1JRg7N/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781955746769051-7083-0/minicluster-data/ts-0/wal/instance:
uuid: "180bb34b014448ffa2daa5f7698fa698"
format_stamp: "Formatted at 2026-06-20 11:46:19 on dist-test-slave-cjc6"
server_key: "913852a14e11691304d0e0210e78bf23"
server_key_iv: "2906dcb1283cc2d1db408b98be06840b"
server_key_version: "encryptionkey@0"
I20260620 11:46:19.826074 26259 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260620 11:46:19.826829 26274 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 11:46:19.826985 26259 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260620 11:46:19.827065 26259 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task1JRg7N/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781955746769051-7083-0/minicluster-data/ts-0/data,/tmp/dist-test-task1JRg7N/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781955746769051-7083-0/minicluster-data/ts-0/wal
uuid: "180bb34b014448ffa2daa5f7698fa698"
format_stamp: "Formatted at 2026-06-20 11:46:19 on dist-test-slave-cjc6"
server_key: "913852a14e11691304d0e0210e78bf23"
server_key_iv: "2906dcb1283cc2d1db408b98be06840b"
server_key_version: "encryptionkey@0"
I20260620 11:46:19.827137 26259 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task1JRg7N/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781955746769051-7083-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-task1JRg7N/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781955746769051-7083-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-task1JRg7N/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781955746769051-7083-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 11:46:19.854630 26259 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 11:46:19.856534 26259 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 11:46:19.856676 26259 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 11:46:19.856952 26259 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 11:46:19.857270 26259 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 11:46:19.857326 26259 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 11:46:19.857362 26259 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 11:46:19.857388 26259 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 11:46:19.863157 26259 rpc_server.cc:307] RPC server started. Bound to: 127.6.234.193:46439
I20260620 11:46:19.863245 26387 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.6.234.193:46439 every 8 connection(s)
I20260620 11:46:19.863567 26259 server_base.cc:1166] Dumped server information to /tmp/dist-test-task1JRg7N/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781955746769051-7083-0/minicluster-data/ts-0/data/info.pb
Jun 20 11:46:19 dist-test-slave-cjc6 krb5kdc[26152](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781955979, etypes {rep=17 tkt=17 ses=17}, kudu/127.6.234.193@KRBTEST.COM for kudu/127.6.234.254@KRBTEST.COM
I20260620 11:46:19.872805 7083 external_mini_cluster.cc:1388] Started /tmp/dist-test-task1JRg7N/build/release/bin/kudu as pid 26259
I20260620 11:46:19.872933 7083 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task1JRg7N/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781955746769051-7083-0/minicluster-data/ts-0/wal/instance
I20260620 11:46:19.873113 7083 external_mini_cluster.cc:1428] Setting key bb12788b643b43392efaca0b24529509
I20260620 11:46:19.873622 26237 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 11:46:19.864327 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.6.234.193:49333 (local address 127.6.234.254:38673)
0620 11:46:19.864397 (+ 70us) server_negotiation.cc:206] Beginning negotiation
0620 11:46:19.864399 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 11:46:19.864595 (+ 196us) server_negotiation.cc:407] Connection header received
0620 11:46:19.865081 (+ 486us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 11:46:19.865082 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 11:46:19.865091 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 11:46:19.865149 (+ 58us) server_negotiation.cc:226] Negotiated authn=SASL
0620 11:46:19.866159 (+ 1010us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 11:46:19.866676 (+ 517us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 11:46:19.867738 (+ 1062us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 11:46:19.867961 (+ 223us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 11:46:19.868029 (+ 68us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 11:46:19.870261 (+ 2232us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 11:46:19.870262 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 11:46:19.870263 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 11:46:19.870270 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0620 11:46:19.871901 (+ 1631us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 11:46:19.872428 (+ 527us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 11:46:19.872429 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 11:46:19.872430 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 11:46:19.872523 (+ 93us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 11:46:19.872904 (+ 381us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 11:46:19.872905 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 11:46:19.872906 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 11:46:19.873049 (+ 143us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 11:46:19.873125 (+ 76us) server_negotiation.cc:1035] Waiting for connection context
0620 11:46:19.873440 (+ 315us) server_negotiation.cc:299] Negotiation successful
0620 11:46:19.873529 (+ 89us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":43}
I20260620 11:46:19.873849 26390 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 11:46:19.864296 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.234.254:38673 (local address 127.6.234.193:49333)
0620 11:46:19.864466 (+ 170us) negotiation.cc:107] Waiting for socket to connect
0620 11:46:19.864511 (+ 45us) client_negotiation.cc:174] Beginning negotiation
0620 11:46:19.864994 (+ 483us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 11:46:19.865202 (+ 208us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 11:46:19.865206 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 11:46:19.865863 (+ 657us) client_negotiation.cc:189] Negotiated authn=SASL
0620 11:46:19.866071 (+ 208us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 11:46:19.866076 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 11:46:19.866777 (+ 701us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 11:46:19.866778 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 11:46:19.867631 (+ 853us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 11:46:19.867633 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 11:46:19.868055 (+ 422us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 11:46:19.868056 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 11:46:19.868156 (+ 100us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 11:46:19.868655 (+ 499us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 11:46:19.868663 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0620 11:46:19.870147 (+ 1484us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 11:46:19.871995 (+ 1848us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 11:46:19.872003 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 11:46:19.872005 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0620 11:46:19.872328 (+ 323us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 11:46:19.872634 (+ 306us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 11:46:19.872635 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 11:46:19.872635 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0620 11:46:19.872818 (+ 183us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 11:46:19.873129 (+ 311us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 11:46:19.873136 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 11:46:19.873280 (+ 144us) client_negotiation.cc:769] Sending connection context
0620 11:46:19.873409 (+ 129us) client_negotiation.cc:240] Negotiation successful
0620 11:46:19.873587 (+ 178us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":127,"thread_start_us":93,"threads_started":1}
I20260620 11:46:19.874291 26388 heartbeater.cc:347] Connected to a master server at 127.6.234.254:38673
I20260620 11:46:19.874388 26388 heartbeater.cc:464] Registering TS with master...
I20260620 11:46:19.874544 26388 heartbeater.cc:511] Master 127.6.234.254:38673 requested a full tablet report, sending...
I20260620 11:46:19.875031 26197 ts_manager.cc:194] Registered new tserver with Master: 180bb34b014448ffa2daa5f7698fa698 (127.6.234.193:46439)
I20260620 11:46:19.875653 26197 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.6.234.193@KRBTEST.COM'} at 127.6.234.193:49333
WARNING: no policy specified for kudu/127.6.234.194@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.6.234.194@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.6.234.194 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task1JRg7N/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781955746769051-7083-0/krb5kdc/kudu_127.6.234.194.keytab.
Entry for principal kudu/127.6.234.194 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task1JRg7N/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781955746769051-7083-0/krb5kdc/kudu_127.6.234.194.keytab.
I20260620 11:46:19.897050 7083 external_mini_cluster.cc:1326] Running /tmp/dist-test-task1JRg7N/build/release/bin/kudu
/tmp/dist-test-task1JRg7N/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task1JRg7N/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781955746769051-7083-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-task1JRg7N/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781955746769051-7083-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task1JRg7N/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781955746769051-7083-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-task1JRg7N/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781955746769051-7083-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.6.234.194:0
--local_ip_for_outbound_sockets=127.6.234.194
--webserver_interface=127.6.234.194
--webserver_port=0
--tserver_master_addrs=127.6.234.254:38673
--keytab_file=/tmp/dist-test-task1JRg7N/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781955746769051-7083-0/krb5kdc/kudu_127.6.234.194.keytab
--principal=kudu/127.6.234.194
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.6.234.212:41711
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task1JRg7N/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781955746769051-7083-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task1JRg7N/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781955746769051-7083-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task1JRg7N/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781955746769051-7083-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 11:46:19.978222 26393 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 11:46:19.978443 26393 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 11:46:19.978469 26393 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 11:46:19.979736 26393 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 11:46:19.979784 26393 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 11:46:19.979813 26393 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.6.234.194
I20260620 11:46:19.981467 26393 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.6.234.212:41711
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task1JRg7N/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781955746769051-7083-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-task1JRg7N/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781955746769051-7083-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task1JRg7N/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781955746769051-7083-0/krb5kdc/kudu_127.6.234.194.keytab
--principal=kudu/127.6.234.194
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.6.234.194:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task1JRg7N/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781955746769051-7083-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.6.234.194
--webserver_port=0
--tserver_master_addrs=127.6.234.254:38673
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.26393
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.6.234.194
--log_dir=/tmp/dist-test-task1JRg7N/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781955746769051-7083-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 20 Jun 2026 10:32:37 UTC on ccycloud.quasar-bgyofv.root.comops.site
build id 4457173
I20260620 11:46:19.981741 26393 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 11:46:19.982014 26393 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260620 11:46:19.987269 26393 server_base.cc:1034] running on GCE node
W20260620 11:46:19.987239 26398 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 11:46:19.987236 26399 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 11:46:19.987303 26401 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 11:46:19.987687 26393 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 11:46:19.987881 26393 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 11:46:19.989028 26393 hybrid_clock.cc:648] HybridClock initialized: now 1781955979989021 us; error 31 us; skew 500 ppm
Jun 20 11:46:19 dist-test-slave-cjc6 krb5kdc[26152](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781955979, etypes {rep=17 tkt=17 ses=17}, kudu/127.6.234.194@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 11:46:19.991463 26393 init.cc:377] Logged in from keytab as kudu/127.6.234.194@KRBTEST.COM (short username kudu)
I20260620 11:46:19.991887 26393 webserver.cc:460] Webserver started at http://127.6.234.194:46147/ using document root <none> and password file <none>
I20260620 11:46:19.992142 26393 fs_manager.cc:359] Metadata directory not provided
I20260620 11:46:19.992187 26393 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 11:46:19.992349 26393 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 11:46:19.993388 26393 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task1JRg7N/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781955746769051-7083-0/minicluster-data/ts-1/data/instance:
uuid: "495d7e3367924702b1ce9ba3ce523b6e"
format_stamp: "Formatted at 2026-06-20 11:46:19 on dist-test-slave-cjc6"
server_key: "e68c79ec0510aabb4b44092fdc0e7e49"
server_key_iv: "9cb1f3faa992dd79a164e054194233e8"
server_key_version: "encryptionkey@0"
I20260620 11:46:19.993814 26393 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task1JRg7N/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781955746769051-7083-0/minicluster-data/ts-1/wal/instance:
uuid: "495d7e3367924702b1ce9ba3ce523b6e"
format_stamp: "Formatted at 2026-06-20 11:46:19 on dist-test-slave-cjc6"
server_key: "e68c79ec0510aabb4b44092fdc0e7e49"
server_key_iv: "9cb1f3faa992dd79a164e054194233e8"
server_key_version: "encryptionkey@0"
I20260620 11:46:19.995097 26393 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.001s
I20260620 11:46:19.995903 26408 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 11:46:19.996116 26393 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.001s
I20260620 11:46:19.996188 26393 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task1JRg7N/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781955746769051-7083-0/minicluster-data/ts-1/data,/tmp/dist-test-task1JRg7N/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781955746769051-7083-0/minicluster-data/ts-1/wal
uuid: "495d7e3367924702b1ce9ba3ce523b6e"
format_stamp: "Formatted at 2026-06-20 11:46:19 on dist-test-slave-cjc6"
server_key: "e68c79ec0510aabb4b44092fdc0e7e49"
server_key_iv: "9cb1f3faa992dd79a164e054194233e8"
server_key_version: "encryptionkey@0"
I20260620 11:46:19.996274 26393 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task1JRg7N/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781955746769051-7083-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-task1JRg7N/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781955746769051-7083-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-task1JRg7N/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781955746769051-7083-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 11:46:20.019798 26393 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 11:46:20.021920 26393 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 11:46:20.022068 26393 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 11:46:20.022315 26393 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 11:46:20.022637 26393 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 11:46:20.022677 26393 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 11:46:20.022720 26393 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 11:46:20.022747 26393 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 11:46:20.028667 26393 rpc_server.cc:307] RPC server started. Bound to: 127.6.234.194:38075
I20260620 11:46:20.028743 26521 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.6.234.194:38075 every 8 connection(s)
I20260620 11:46:20.029083 26393 server_base.cc:1166] Dumped server information to /tmp/dist-test-task1JRg7N/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781955746769051-7083-0/minicluster-data/ts-1/data/info.pb
I20260620 11:46:20.031885 7083 external_mini_cluster.cc:1388] Started /tmp/dist-test-task1JRg7N/build/release/bin/kudu as pid 26393
I20260620 11:46:20.032027 7083 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task1JRg7N/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781955746769051-7083-0/minicluster-data/ts-1/wal/instance
I20260620 11:46:20.032245 7083 external_mini_cluster.cc:1428] Setting key cca653c62f3a8091616e2305f6245463
Jun 20 11:46:20 dist-test-slave-cjc6 krb5kdc[26152](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781955979, etypes {rep=17 tkt=17 ses=17}, kudu/127.6.234.194@KRBTEST.COM for kudu/127.6.234.254@KRBTEST.COM
I20260620 11:46:20.040478 26237 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 11:46:20.029851 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.6.234.194:34653 (local address 127.6.234.254:38673)
0620 11:46:20.029895 (+ 44us) server_negotiation.cc:206] Beginning negotiation
0620 11:46:20.029897 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 11:46:20.030091 (+ 194us) server_negotiation.cc:407] Connection header received
0620 11:46:20.030594 (+ 503us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 11:46:20.030595 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 11:46:20.030623 (+ 28us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 11:46:20.030697 (+ 74us) server_negotiation.cc:226] Negotiated authn=SASL
0620 11:46:20.031719 (+ 1022us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 11:46:20.032532 (+ 813us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 11:46:20.034015 (+ 1483us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 11:46:20.034204 (+ 189us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 11:46:20.034263 (+ 59us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 11:46:20.036770 (+ 2507us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 11:46:20.036772 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 11:46:20.036773 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 11:46:20.036781 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0620 11:46:20.038753 (+ 1972us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 11:46:20.039204 (+ 451us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 11:46:20.039205 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 11:46:20.039206 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 11:46:20.039278 (+ 72us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 11:46:20.039532 (+ 254us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 11:46:20.039533 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 11:46:20.039533 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 11:46:20.039690 (+ 157us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 11:46:20.039776 (+ 86us) server_negotiation.cc:1035] Waiting for connection context
0620 11:46:20.040250 (+ 474us) server_negotiation.cc:299] Negotiation successful
0620 11:46:20.040347 (+ 97us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":20}
I20260620 11:46:20.040511 26524 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 11:46:20.029833 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.234.254:38673 (local address 127.6.234.194:34653)
0620 11:46:20.029982 (+ 149us) negotiation.cc:107] Waiting for socket to connect
0620 11:46:20.030023 (+ 41us) client_negotiation.cc:174] Beginning negotiation
0620 11:46:20.030475 (+ 452us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 11:46:20.030756 (+ 281us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 11:46:20.030760 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 11:46:20.031388 (+ 628us) client_negotiation.cc:189] Negotiated authn=SASL
0620 11:46:20.031617 (+ 229us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 11:46:20.031622 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 11:46:20.032654 (+ 1032us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 11:46:20.032655 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 11:46:20.033924 (+ 1269us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 11:46:20.033927 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 11:46:20.034292 (+ 365us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 11:46:20.034293 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 11:46:20.034458 (+ 165us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 11:46:20.035071 (+ 613us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 11:46:20.035080 (+ 9us) client_negotiation.cc:656] calling sasl_client_start()
0620 11:46:20.036657 (+ 1577us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 11:46:20.038868 (+ 2211us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 11:46:20.038875 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 11:46:20.038877 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0620 11:46:20.039119 (+ 242us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 11:46:20.039355 (+ 236us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 11:46:20.039356 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 11:46:20.039356 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0620 11:46:20.039461 (+ 105us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 11:46:20.039954 (+ 493us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 11:46:20.039960 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 11:46:20.040096 (+ 136us) client_negotiation.cc:769] Sending connection context
0620 11:46:20.040196 (+ 100us) client_negotiation.cc:240] Negotiation successful
0620 11:46:20.040326 (+ 130us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":111,"thread_start_us":86,"threads_started":1}
I20260620 11:46:20.041121 26522 heartbeater.cc:347] Connected to a master server at 127.6.234.254:38673
I20260620 11:46:20.041229 26522 heartbeater.cc:464] Registering TS with master...
WARNING: no policy specified for kudu/127.6.234.195@KRBTEST.COM; defaulting to no policy
I20260620 11:46:20.041430 26522 heartbeater.cc:511] Master 127.6.234.254:38673 requested a full tablet report, sending...
I20260620 11:46:20.041848 26197 ts_manager.cc:194] Registered new tserver with Master: 495d7e3367924702b1ce9ba3ce523b6e (127.6.234.194:38075)
I20260620 11:46:20.042243 26197 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.6.234.194@KRBTEST.COM'} at 127.6.234.194:34653
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.6.234.195@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.6.234.195 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task1JRg7N/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781955746769051-7083-0/krb5kdc/kudu_127.6.234.195.keytab.
Entry for principal kudu/127.6.234.195 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task1JRg7N/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781955746769051-7083-0/krb5kdc/kudu_127.6.234.195.keytab.
I20260620 11:46:20.060029 7083 external_mini_cluster.cc:1326] Running /tmp/dist-test-task1JRg7N/build/release/bin/kudu
/tmp/dist-test-task1JRg7N/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task1JRg7N/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781955746769051-7083-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-task1JRg7N/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781955746769051-7083-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task1JRg7N/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781955746769051-7083-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-task1JRg7N/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781955746769051-7083-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.6.234.195:0
--local_ip_for_outbound_sockets=127.6.234.195
--webserver_interface=127.6.234.195
--webserver_port=0
--tserver_master_addrs=127.6.234.254:38673
--keytab_file=/tmp/dist-test-task1JRg7N/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781955746769051-7083-0/krb5kdc/kudu_127.6.234.195.keytab
--principal=kudu/127.6.234.195
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.6.234.212:41711
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task1JRg7N/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781955746769051-7083-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task1JRg7N/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781955746769051-7083-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task1JRg7N/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781955746769051-7083-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 11:46:20.139706 26527 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 11:46:20.139912 26527 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 11:46:20.139936 26527 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 11:46:20.140993 26527 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 11:46:20.141026 26527 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 11:46:20.141052 26527 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.6.234.195
I20260620 11:46:20.142433 26527 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.6.234.212:41711
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task1JRg7N/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781955746769051-7083-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-task1JRg7N/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781955746769051-7083-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task1JRg7N/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781955746769051-7083-0/krb5kdc/kudu_127.6.234.195.keytab
--principal=kudu/127.6.234.195
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.6.234.195:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task1JRg7N/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781955746769051-7083-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.6.234.195
--webserver_port=0
--tserver_master_addrs=127.6.234.254:38673
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.26527
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.6.234.195
--log_dir=/tmp/dist-test-task1JRg7N/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781955746769051-7083-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 20 Jun 2026 10:32:37 UTC on ccycloud.quasar-bgyofv.root.comops.site
build id 4457173
I20260620 11:46:20.142619 26527 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 11:46:20.142783 26527 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 11:46:20.147632 26533 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 11:46:20.147621 26532 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 11:46:20.147655 26535 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 11:46:20.147908 26527 server_base.cc:1034] running on GCE node
I20260620 11:46:20.148119 26527 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 11:46:20.148351 26527 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 11:46:20.149523 26527 hybrid_clock.cc:648] HybridClock initialized: now 1781955980149509 us; error 25 us; skew 500 ppm
Jun 20 11:46:20 dist-test-slave-cjc6 krb5kdc[26152](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781955980, etypes {rep=17 tkt=17 ses=17}, kudu/127.6.234.195@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 11:46:20.152010 26527 init.cc:377] Logged in from keytab as kudu/127.6.234.195@KRBTEST.COM (short username kudu)
I20260620 11:46:20.152486 26527 webserver.cc:460] Webserver started at http://127.6.234.195:38271/ using document root <none> and password file <none>
I20260620 11:46:20.152721 26527 fs_manager.cc:359] Metadata directory not provided
I20260620 11:46:20.152791 26527 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 11:46:20.152925 26527 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 11:46:20.154036 26527 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task1JRg7N/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781955746769051-7083-0/minicluster-data/ts-2/data/instance:
uuid: "0142ad50ae1a4d429b744b5731e6a340"
format_stamp: "Formatted at 2026-06-20 11:46:20 on dist-test-slave-cjc6"
server_key: "7c63451aee9571cd8627fc97d7aa58fd"
server_key_iv: "58080c674c48b07fab283a5a08355f0f"
server_key_version: "encryptionkey@0"
I20260620 11:46:20.154412 26527 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task1JRg7N/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781955746769051-7083-0/minicluster-data/ts-2/wal/instance:
uuid: "0142ad50ae1a4d429b744b5731e6a340"
format_stamp: "Formatted at 2026-06-20 11:46:20 on dist-test-slave-cjc6"
server_key: "7c63451aee9571cd8627fc97d7aa58fd"
server_key_iv: "58080c674c48b07fab283a5a08355f0f"
server_key_version: "encryptionkey@0"
I20260620 11:46:20.155750 26527 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.001s sys 0.000s
I20260620 11:46:20.156514 26542 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 11:46:20.156735 26527 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260620 11:46:20.156860 26527 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task1JRg7N/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781955746769051-7083-0/minicluster-data/ts-2/data,/tmp/dist-test-task1JRg7N/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781955746769051-7083-0/minicluster-data/ts-2/wal
uuid: "0142ad50ae1a4d429b744b5731e6a340"
format_stamp: "Formatted at 2026-06-20 11:46:20 on dist-test-slave-cjc6"
server_key: "7c63451aee9571cd8627fc97d7aa58fd"
server_key_iv: "58080c674c48b07fab283a5a08355f0f"
server_key_version: "encryptionkey@0"
I20260620 11:46:20.156949 26527 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task1JRg7N/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781955746769051-7083-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-task1JRg7N/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781955746769051-7083-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-task1JRg7N/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781955746769051-7083-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 11:46:20.182757 26527 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 11:46:20.184723 26527 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 11:46:20.184890 26527 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 11:46:20.185133 26527 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 11:46:20.185467 26527 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 11:46:20.185527 26527 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 11:46:20.185570 26527 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 11:46:20.185596 26527 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 11:46:20.191550 26527 rpc_server.cc:307] RPC server started. Bound to: 127.6.234.195:44799
I20260620 11:46:20.191605 26655 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.6.234.195:44799 every 8 connection(s)
I20260620 11:46:20.191939 26527 server_base.cc:1166] Dumped server information to /tmp/dist-test-task1JRg7N/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781955746769051-7083-0/minicluster-data/ts-2/data/info.pb
I20260620 11:46:20.194867 7083 external_mini_cluster.cc:1388] Started /tmp/dist-test-task1JRg7N/build/release/bin/kudu as pid 26527
I20260620 11:46:20.194974 7083 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task1JRg7N/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781955746769051-7083-0/minicluster-data/ts-2/wal/instance
I20260620 11:46:20.195171 7083 external_mini_cluster.cc:1428] Setting key 56496f30c4bf5be7ac0dd6bdfd8072d7
Jun 20 11:46:20 dist-test-slave-cjc6 krb5kdc[26152](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781955980, etypes {rep=17 tkt=17 ses=17}, kudu/127.6.234.195@KRBTEST.COM for kudu/127.6.234.254@KRBTEST.COM
I20260620 11:46:20.201215 26237 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 11:46:20.192634 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.6.234.195:47113 (local address 127.6.234.254:38673)
0620 11:46:20.192684 (+ 50us) server_negotiation.cc:206] Beginning negotiation
0620 11:46:20.192686 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 11:46:20.192903 (+ 217us) server_negotiation.cc:407] Connection header received
0620 11:46:20.193352 (+ 449us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 11:46:20.193353 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 11:46:20.193364 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 11:46:20.193417 (+ 53us) server_negotiation.cc:226] Negotiated authn=SASL
0620 11:46:20.194375 (+ 958us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 11:46:20.195034 (+ 659us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 11:46:20.196340 (+ 1306us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 11:46:20.196527 (+ 187us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 11:46:20.196583 (+ 56us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 11:46:20.198671 (+ 2088us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 11:46:20.198672 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 11:46:20.198672 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 11:46:20.198679 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0620 11:46:20.200133 (+ 1454us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 11:46:20.200485 (+ 352us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 11:46:20.200486 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 11:46:20.200486 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 11:46:20.200535 (+ 49us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 11:46:20.200738 (+ 203us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 11:46:20.200738 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 11:46:20.200739 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 11:46:20.200831 (+ 92us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 11:46:20.200880 (+ 49us) server_negotiation.cc:1035] Waiting for connection context
0620 11:46:20.201098 (+ 218us) server_negotiation.cc:299] Negotiation successful
0620 11:46:20.201152 (+ 54us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":25}
I20260620 11:46:20.201442 26658 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 11:46:20.192629 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.234.254:38673 (local address 127.6.234.195:47113)
0620 11:46:20.192781 (+ 152us) negotiation.cc:107] Waiting for socket to connect
0620 11:46:20.192815 (+ 34us) client_negotiation.cc:174] Beginning negotiation
0620 11:46:20.193273 (+ 458us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 11:46:20.193471 (+ 198us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 11:46:20.193474 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 11:46:20.194110 (+ 636us) client_negotiation.cc:189] Negotiated authn=SASL
0620 11:46:20.194301 (+ 191us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 11:46:20.194305 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 11:46:20.195126 (+ 821us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 11:46:20.195127 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 11:46:20.196270 (+ 1143us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 11:46:20.196273 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 11:46:20.196600 (+ 327us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 11:46:20.196601 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 11:46:20.196695 (+ 94us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 11:46:20.197140 (+ 445us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 11:46:20.197148 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0620 11:46:20.198590 (+ 1442us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 11:46:20.200207 (+ 1617us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 11:46:20.200211 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 11:46:20.200213 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0620 11:46:20.200425 (+ 212us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 11:46:20.200591 (+ 166us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 11:46:20.200592 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 11:46:20.200592 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0620 11:46:20.200685 (+ 93us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 11:46:20.200883 (+ 198us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 11:46:20.200887 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 11:46:20.200997 (+ 110us) client_negotiation.cc:769] Sending connection context
0620 11:46:20.201078 (+ 81us) client_negotiation.cc:240] Negotiation successful
0620 11:46:20.201189 (+ 111us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":116,"thread_start_us":89,"threads_started":1}
I20260620 11:46:20.201959 26656 heartbeater.cc:347] Connected to a master server at 127.6.234.254:38673
I20260620 11:46:20.202039 26656 heartbeater.cc:464] Registering TS with master...
I20260620 11:46:20.202225 26656 heartbeater.cc:511] Master 127.6.234.254:38673 requested a full tablet report, sending...
I20260620 11:46:20.202615 26197 ts_manager.cc:194] Registered new tserver with Master: 0142ad50ae1a4d429b744b5731e6a340 (127.6.234.195:44799)
I20260620 11:46:20.203020 26197 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.6.234.195@KRBTEST.COM'} at 127.6.234.195:47113
I20260620 11:46:20.206967 7083 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260620 11:46:20.214428 26237 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 11:46:20.208232 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41538 (local address 127.6.234.254:38673)
0620 11:46:20.208321 (+ 89us) server_negotiation.cc:206] Beginning negotiation
0620 11:46:20.208323 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 11:46:20.208425 (+ 102us) server_negotiation.cc:407] Connection header received
0620 11:46:20.208501 (+ 76us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 11:46:20.208502 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 11:46:20.208512 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 11:46:20.208599 (+ 87us) server_negotiation.cc:226] Negotiated authn=SASL
0620 11:46:20.209145 (+ 546us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 11:46:20.209647 (+ 502us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 11:46:20.210417 (+ 770us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 11:46:20.210610 (+ 193us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 11:46:20.210685 (+ 75us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 11:46:20.211852 (+ 1167us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 11:46:20.211854 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 11:46:20.211854 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 11:46:20.211861 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0620 11:46:20.213146 (+ 1285us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 11:46:20.213506 (+ 360us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 11:46:20.213507 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 11:46:20.213508 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 11:46:20.213566 (+ 58us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 11:46:20.213818 (+ 252us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 11:46:20.213819 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 11:46:20.213820 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 11:46:20.213984 (+ 164us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 11:46:20.214094 (+ 110us) server_negotiation.cc:1035] Waiting for connection context
0620 11:46:20.214196 (+ 102us) server_negotiation.cc:299] Negotiation successful
0620 11:46:20.214264 (+ 68us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":65}
I20260620 11:46:20.216157 26197 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:41538:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260620 11:46:20.217753 26197 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260620 11:46:20.224074 26668 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 11:46:20.220015 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.234.193:46439 (local address 127.0.0.1:48060)
0620 11:46:20.220676 (+ 661us) negotiation.cc:107] Waiting for socket to connect
0620 11:46:20.220686 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0620 11:46:20.220761 (+ 75us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 11:46:20.221032 (+ 271us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 11:46:20.221032 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 11:46:20.221034 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 11:46:20.221065 (+ 31us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 11:46:20.221066 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 11:46:20.222019 (+ 953us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 11:46:20.222020 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 11:46:20.223053 (+ 1033us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 11:46:20.223055 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 11:46:20.223759 (+ 704us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 11:46:20.223760 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 11:46:20.223836 (+ 76us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 11:46:20.223850 (+ 14us) client_negotiation.cc:769] Sending connection context
0620 11:46:20.223948 (+ 98us) client_negotiation.cc:240] Negotiation successful
0620 11:46:20.224003 (+ 55us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":643,"mutex_wait_us":2,"thread_start_us":83,"threads_started":1}
I20260620 11:46:20.224092 26672 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 11:46:20.220242 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48060 (local address 127.6.234.193:46439)
0620 11:46:20.220628 (+ 386us) server_negotiation.cc:206] Beginning negotiation
0620 11:46:20.220631 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0620 11:46:20.220735 (+ 104us) server_negotiation.cc:407] Connection header received
0620 11:46:20.220856 (+ 121us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 11:46:20.220858 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 11:46:20.220874 (+ 16us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 11:46:20.220927 (+ 53us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 11:46:20.221139 (+ 212us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 11:46:20.221887 (+ 748us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 11:46:20.223137 (+ 1250us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 11:46:20.223666 (+ 529us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 11:46:20.223747 (+ 81us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 11:46:20.223804 (+ 57us) server_negotiation.cc:1035] Waiting for connection context
0620 11:46:20.223973 (+ 169us) server_negotiation.cc:299] Negotiation successful
0620 11:46:20.224011 (+ 38us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":346,"thread_start_us":76,"threads_started":1}
I20260620 11:46:20.224534 26669 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 11:46:20.219986 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.234.195:44799 (local address 127.0.0.1:46162)
0620 11:46:20.220403 (+ 417us) negotiation.cc:107] Waiting for socket to connect
0620 11:46:20.220413 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0620 11:46:20.220490 (+ 77us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 11:46:20.220746 (+ 256us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 11:46:20.220746 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 11:46:20.220748 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 11:46:20.220791 (+ 43us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 11:46:20.220792 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 11:46:20.222048 (+ 1256us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 11:46:20.222049 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 11:46:20.223329 (+ 1280us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 11:46:20.223354 (+ 25us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 11:46:20.224305 (+ 951us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 11:46:20.224306 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 11:46:20.224363 (+ 57us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 11:46:20.224366 (+ 3us) client_negotiation.cc:769] Sending connection context
0620 11:46:20.224410 (+ 44us) client_negotiation.cc:240] Negotiation successful
0620 11:46:20.224450 (+ 40us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":397,"mutex_wait_us":91,"thread_start_us":74,"threads_started":1}
I20260620 11:46:20.224761 26667 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 11:46:20.219986 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.234.194:38075 (local address 127.0.0.1:51466)
0620 11:46:20.220147 (+ 161us) negotiation.cc:107] Waiting for socket to connect
0620 11:46:20.220186 (+ 39us) client_negotiation.cc:174] Beginning negotiation
0620 11:46:20.220250 (+ 64us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 11:46:20.220564 (+ 314us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 11:46:20.220567 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 11:46:20.220570 (+ 3us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 11:46:20.220641 (+ 71us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 11:46:20.220643 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 11:46:20.222022 (+ 1379us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 11:46:20.222023 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 11:46:20.223340 (+ 1317us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 11:46:20.223343 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 11:46:20.224549 (+ 1206us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 11:46:20.224550 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 11:46:20.224618 (+ 68us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 11:46:20.224622 (+ 4us) client_negotiation.cc:769] Sending connection context
0620 11:46:20.224663 (+ 41us) client_negotiation.cc:240] Negotiation successful
0620 11:46:20.224699 (+ 36us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":137,"mutex_wait_us":66,"thread_start_us":60,"threads_started":1}
I20260620 11:46:20.224815 26670 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 11:46:20.220169 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46162 (local address 127.6.234.195:44799)
0620 11:46:20.220358 (+ 189us) server_negotiation.cc:206] Beginning negotiation
0620 11:46:20.220363 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 11:46:20.220483 (+ 120us) server_negotiation.cc:407] Connection header received
0620 11:46:20.220566 (+ 83us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 11:46:20.220568 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 11:46:20.220583 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 11:46:20.220669 (+ 86us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 11:46:20.220863 (+ 194us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 11:46:20.221940 (+ 1077us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 11:46:20.223439 (+ 1499us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 11:46:20.224141 (+ 702us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 11:46:20.224216 (+ 75us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 11:46:20.224274 (+ 58us) server_negotiation.cc:1035] Waiting for connection context
0620 11:46:20.224694 (+ 420us) server_negotiation.cc:299] Negotiation successful
0620 11:46:20.224735 (+ 41us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":162,"thread_start_us":88,"threads_started":1}
I20260620 11:46:20.224857 26322 tablet_service.cc:1467] Processing CreateTablet for tablet 61ce3f703e8d4ece9d1adce07d3f7ad5 (DEFAULT_TABLE table=test-table [id=9b299d84348d43daaa66453804ddd4a3]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 11:46:20.225112 26671 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 11:46:20.220197 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51466 (local address 127.6.234.194:38075)
0620 11:46:20.220391 (+ 194us) server_negotiation.cc:206] Beginning negotiation
0620 11:46:20.220396 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 11:46:20.220407 (+ 11us) server_negotiation.cc:407] Connection header received
0620 11:46:20.220448 (+ 41us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 11:46:20.220450 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 11:46:20.220466 (+ 16us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 11:46:20.220549 (+ 83us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 11:46:20.220733 (+ 184us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 11:46:20.221907 (+ 1174us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 11:46:20.223439 (+ 1532us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 11:46:20.224459 (+ 1020us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 11:46:20.224529 (+ 70us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 11:46:20.224751 (+ 222us) server_negotiation.cc:1035] Waiting for connection context
0620 11:46:20.224775 (+ 24us) server_negotiation.cc:299] Negotiation successful
0620 11:46:20.225036 (+ 261us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":162,"thread_start_us":83,"threads_started":1}
I20260620 11:46:20.225260 26322 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 61ce3f703e8d4ece9d1adce07d3f7ad5. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 11:46:20.225422 26590 tablet_service.cc:1467] Processing CreateTablet for tablet 61ce3f703e8d4ece9d1adce07d3f7ad5 (DEFAULT_TABLE table=test-table [id=9b299d84348d43daaa66453804ddd4a3]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 11:46:20.225764 26590 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 61ce3f703e8d4ece9d1adce07d3f7ad5. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 11:46:20.225685 26456 tablet_service.cc:1467] Processing CreateTablet for tablet 61ce3f703e8d4ece9d1adce07d3f7ad5 (DEFAULT_TABLE table=test-table [id=9b299d84348d43daaa66453804ddd4a3]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 11:46:20.225993 26456 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 61ce3f703e8d4ece9d1adce07d3f7ad5. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 11:46:20.228283 26673 tablet_bootstrap.cc:492] T 61ce3f703e8d4ece9d1adce07d3f7ad5 P 180bb34b014448ffa2daa5f7698fa698: Bootstrap starting.
I20260620 11:46:20.228969 26673 tablet_bootstrap.cc:654] T 61ce3f703e8d4ece9d1adce07d3f7ad5 P 180bb34b014448ffa2daa5f7698fa698: Neither blocks nor log segments found. Creating new log.
I20260620 11:46:20.228982 26674 tablet_bootstrap.cc:492] T 61ce3f703e8d4ece9d1adce07d3f7ad5 P 0142ad50ae1a4d429b744b5731e6a340: Bootstrap starting.
I20260620 11:46:20.229722 26674 tablet_bootstrap.cc:654] T 61ce3f703e8d4ece9d1adce07d3f7ad5 P 0142ad50ae1a4d429b744b5731e6a340: Neither blocks nor log segments found. Creating new log.
I20260620 11:46:20.229687 26673 log.cc:824] T 61ce3f703e8d4ece9d1adce07d3f7ad5 P 180bb34b014448ffa2daa5f7698fa698: Log is configured to *not* fsync() on all Append() calls
I20260620 11:46:20.229856 26675 tablet_bootstrap.cc:492] T 61ce3f703e8d4ece9d1adce07d3f7ad5 P 495d7e3367924702b1ce9ba3ce523b6e: Bootstrap starting.
I20260620 11:46:20.230310 26674 log.cc:824] T 61ce3f703e8d4ece9d1adce07d3f7ad5 P 0142ad50ae1a4d429b744b5731e6a340: Log is configured to *not* fsync() on all Append() calls
I20260620 11:46:20.230559 26675 tablet_bootstrap.cc:654] T 61ce3f703e8d4ece9d1adce07d3f7ad5 P 495d7e3367924702b1ce9ba3ce523b6e: Neither blocks nor log segments found. Creating new log.
I20260620 11:46:20.230620 26673 tablet_bootstrap.cc:492] T 61ce3f703e8d4ece9d1adce07d3f7ad5 P 180bb34b014448ffa2daa5f7698fa698: No bootstrap required, opened a new log
I20260620 11:46:20.230682 26673 ts_tablet_manager.cc:1397] T 61ce3f703e8d4ece9d1adce07d3f7ad5 P 180bb34b014448ffa2daa5f7698fa698: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20260620 11:46:20.230929 26674 tablet_bootstrap.cc:492] T 61ce3f703e8d4ece9d1adce07d3f7ad5 P 0142ad50ae1a4d429b744b5731e6a340: No bootstrap required, opened a new log
I20260620 11:46:20.231024 26674 ts_tablet_manager.cc:1397] T 61ce3f703e8d4ece9d1adce07d3f7ad5 P 0142ad50ae1a4d429b744b5731e6a340: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20260620 11:46:20.231109 26675 log.cc:824] T 61ce3f703e8d4ece9d1adce07d3f7ad5 P 495d7e3367924702b1ce9ba3ce523b6e: Log is configured to *not* fsync() on all Append() calls
I20260620 11:46:20.232314 26673 raft_consensus.cc:348] T 61ce3f703e8d4ece9d1adce07d3f7ad5 P 180bb34b014448ffa2daa5f7698fa698 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "495d7e3367924702b1ce9ba3ce523b6e" member_type: VOTER last_known_addr { host: "127.6.234.194" port: 38075 } } peers { permanent_uuid: "0142ad50ae1a4d429b744b5731e6a340" member_type: VOTER last_known_addr { host: "127.6.234.195" port: 44799 } } peers { permanent_uuid: "180bb34b014448ffa2daa5f7698fa698" member_type: VOTER last_known_addr { host: "127.6.234.193" port: 46439 } }
I20260620 11:46:20.232442 26673 raft_consensus.cc:374] T 61ce3f703e8d4ece9d1adce07d3f7ad5 P 180bb34b014448ffa2daa5f7698fa698 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 11:46:20.232462 26673 raft_consensus.cc:729] T 61ce3f703e8d4ece9d1adce07d3f7ad5 P 180bb34b014448ffa2daa5f7698fa698 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 180bb34b014448ffa2daa5f7698fa698, State: Initialized, Role: FOLLOWER
I20260620 11:46:20.232550 26673 consensus_queue.cc:260] T 61ce3f703e8d4ece9d1adce07d3f7ad5 P 180bb34b014448ffa2daa5f7698fa698 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "495d7e3367924702b1ce9ba3ce523b6e" member_type: VOTER last_known_addr { host: "127.6.234.194" port: 38075 } } peers { permanent_uuid: "0142ad50ae1a4d429b744b5731e6a340" member_type: VOTER last_known_addr { host: "127.6.234.195" port: 44799 } } peers { permanent_uuid: "180bb34b014448ffa2daa5f7698fa698" member_type: VOTER last_known_addr { host: "127.6.234.193" port: 46439 } }
I20260620 11:46:20.232522 26674 raft_consensus.cc:348] T 61ce3f703e8d4ece9d1adce07d3f7ad5 P 0142ad50ae1a4d429b744b5731e6a340 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "495d7e3367924702b1ce9ba3ce523b6e" member_type: VOTER last_known_addr { host: "127.6.234.194" port: 38075 } } peers { permanent_uuid: "0142ad50ae1a4d429b744b5731e6a340" member_type: VOTER last_known_addr { host: "127.6.234.195" port: 44799 } } peers { permanent_uuid: "180bb34b014448ffa2daa5f7698fa698" member_type: VOTER last_known_addr { host: "127.6.234.193" port: 46439 } }
I20260620 11:46:20.232715 26674 raft_consensus.cc:374] T 61ce3f703e8d4ece9d1adce07d3f7ad5 P 0142ad50ae1a4d429b744b5731e6a340 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 11:46:20.232739 26673 ts_tablet_manager.cc:1428] T 61ce3f703e8d4ece9d1adce07d3f7ad5 P 180bb34b014448ffa2daa5f7698fa698: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20260620 11:46:20.232757 26674 raft_consensus.cc:729] T 61ce3f703e8d4ece9d1adce07d3f7ad5 P 0142ad50ae1a4d429b744b5731e6a340 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0142ad50ae1a4d429b744b5731e6a340, State: Initialized, Role: FOLLOWER
I20260620 11:46:20.232829 26388 heartbeater.cc:503] Master 127.6.234.254:38673 was elected leader, sending a full tablet report...
I20260620 11:46:20.232864 26674 consensus_queue.cc:260] T 61ce3f703e8d4ece9d1adce07d3f7ad5 P 0142ad50ae1a4d429b744b5731e6a340 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "495d7e3367924702b1ce9ba3ce523b6e" member_type: VOTER last_known_addr { host: "127.6.234.194" port: 38075 } } peers { permanent_uuid: "0142ad50ae1a4d429b744b5731e6a340" member_type: VOTER last_known_addr { host: "127.6.234.195" port: 44799 } } peers { permanent_uuid: "180bb34b014448ffa2daa5f7698fa698" member_type: VOTER last_known_addr { host: "127.6.234.193" port: 46439 } }
I20260620 11:46:20.233153 26674 ts_tablet_manager.cc:1428] T 61ce3f703e8d4ece9d1adce07d3f7ad5 P 0142ad50ae1a4d429b744b5731e6a340: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20260620 11:46:20.233186 26656 heartbeater.cc:503] Master 127.6.234.254:38673 was elected leader, sending a full tablet report...
I20260620 11:46:20.233744 26675 tablet_bootstrap.cc:492] T 61ce3f703e8d4ece9d1adce07d3f7ad5 P 495d7e3367924702b1ce9ba3ce523b6e: No bootstrap required, opened a new log
I20260620 11:46:20.233814 26675 ts_tablet_manager.cc:1397] T 61ce3f703e8d4ece9d1adce07d3f7ad5 P 495d7e3367924702b1ce9ba3ce523b6e: Time spent bootstrapping tablet: real 0.004s user 0.002s sys 0.000s
I20260620 11:46:20.235364 26675 raft_consensus.cc:348] T 61ce3f703e8d4ece9d1adce07d3f7ad5 P 495d7e3367924702b1ce9ba3ce523b6e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "495d7e3367924702b1ce9ba3ce523b6e" member_type: VOTER last_known_addr { host: "127.6.234.194" port: 38075 } } peers { permanent_uuid: "0142ad50ae1a4d429b744b5731e6a340" member_type: VOTER last_known_addr { host: "127.6.234.195" port: 44799 } } peers { permanent_uuid: "180bb34b014448ffa2daa5f7698fa698" member_type: VOTER last_known_addr { host: "127.6.234.193" port: 46439 } }
I20260620 11:46:20.235525 26675 raft_consensus.cc:374] T 61ce3f703e8d4ece9d1adce07d3f7ad5 P 495d7e3367924702b1ce9ba3ce523b6e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 11:46:20.235572 26675 raft_consensus.cc:729] T 61ce3f703e8d4ece9d1adce07d3f7ad5 P 495d7e3367924702b1ce9ba3ce523b6e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 495d7e3367924702b1ce9ba3ce523b6e, State: Initialized, Role: FOLLOWER
I20260620 11:46:20.235678 26675 consensus_queue.cc:260] T 61ce3f703e8d4ece9d1adce07d3f7ad5 P 495d7e3367924702b1ce9ba3ce523b6e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "495d7e3367924702b1ce9ba3ce523b6e" member_type: VOTER last_known_addr { host: "127.6.234.194" port: 38075 } } peers { permanent_uuid: "0142ad50ae1a4d429b744b5731e6a340" member_type: VOTER last_known_addr { host: "127.6.234.195" port: 44799 } } peers { permanent_uuid: "180bb34b014448ffa2daa5f7698fa698" member_type: VOTER last_known_addr { host: "127.6.234.193" port: 46439 } }
I20260620 11:46:20.235941 26675 ts_tablet_manager.cc:1428] T 61ce3f703e8d4ece9d1adce07d3f7ad5 P 495d7e3367924702b1ce9ba3ce523b6e: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20260620 11:46:20.235970 26522 heartbeater.cc:503] Master 127.6.234.254:38673 was elected leader, sending a full tablet report...
W20260620 11:46:20.279716 26523 tablet.cc:2369] T 61ce3f703e8d4ece9d1adce07d3f7ad5 P 495d7e3367924702b1ce9ba3ce523b6e: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260620 11:46:20.364363 26389 tablet.cc:2369] T 61ce3f703e8d4ece9d1adce07d3f7ad5 P 180bb34b014448ffa2daa5f7698fa698: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260620 11:46:20.381230 26679 raft_consensus.cc:482] T 61ce3f703e8d4ece9d1adce07d3f7ad5 P 180bb34b014448ffa2daa5f7698fa698 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260620 11:46:20.381407 26679 raft_consensus.cc:504] T 61ce3f703e8d4ece9d1adce07d3f7ad5 P 180bb34b014448ffa2daa5f7698fa698 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "495d7e3367924702b1ce9ba3ce523b6e" member_type: VOTER last_known_addr { host: "127.6.234.194" port: 38075 } } peers { permanent_uuid: "0142ad50ae1a4d429b744b5731e6a340" member_type: VOTER last_known_addr { host: "127.6.234.195" port: 44799 } } peers { permanent_uuid: "180bb34b014448ffa2daa5f7698fa698" member_type: VOTER last_known_addr { host: "127.6.234.193" port: 46439 } }
I20260620 11:46:20.381853 26679 leader_election.cc:288] T 61ce3f703e8d4ece9d1adce07d3f7ad5 P 180bb34b014448ffa2daa5f7698fa698 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 495d7e3367924702b1ce9ba3ce523b6e (127.6.234.194:38075), 0142ad50ae1a4d429b744b5731e6a340 (127.6.234.195:44799)
I20260620 11:46:20.384907 26670 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 11:46:20.382193 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.6.234.193:36881 (local address 127.6.234.195:44799)
0620 11:46:20.382246 (+ 53us) server_negotiation.cc:206] Beginning negotiation
0620 11:46:20.382249 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0620 11:46:20.382361 (+ 112us) server_negotiation.cc:407] Connection header received
0620 11:46:20.382434 (+ 73us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 11:46:20.382436 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 11:46:20.382446 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 11:46:20.382502 (+ 56us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 11:46:20.382709 (+ 207us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 11:46:20.383162 (+ 453us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 11:46:20.384156 (+ 994us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 11:46:20.384556 (+ 400us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 11:46:20.384614 (+ 58us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 11:46:20.384622 (+ 8us) server_negotiation.cc:1035] Waiting for connection context
0620 11:46:20.384780 (+ 158us) server_negotiation.cc:299] Negotiation successful
0620 11:46:20.384824 (+ 44us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":30}
I20260620 11:46:20.384939 26683 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 11:46:20.382104 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.234.195:44799 (local address 127.6.234.193:36881)
0620 11:46:20.382275 (+ 171us) negotiation.cc:107] Waiting for socket to connect
0620 11:46:20.382287 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0620 11:46:20.382351 (+ 64us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 11:46:20.382519 (+ 168us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 11:46:20.382519 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 11:46:20.382521 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 11:46:20.382627 (+ 106us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 11:46:20.382628 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 11:46:20.383276 (+ 648us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 11:46:20.383277 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 11:46:20.384068 (+ 791us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 11:46:20.384069 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 11:46:20.384636 (+ 567us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 11:46:20.384637 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 11:46:20.384708 (+ 71us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 11:46:20.384712 (+ 4us) client_negotiation.cc:769] Sending connection context
0620 11:46:20.384770 (+ 58us) client_negotiation.cc:240] Negotiation successful
0620 11:46:20.384823 (+ 53us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":138,"thread_start_us":60,"threads_started":1}
I20260620 11:46:20.384949 26671 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 11:46:20.382105 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.6.234.193:47379 (local address 127.6.234.194:38075)
0620 11:46:20.382156 (+ 51us) server_negotiation.cc:206] Beginning negotiation
0620 11:46:20.382158 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 11:46:20.382287 (+ 129us) server_negotiation.cc:407] Connection header received
0620 11:46:20.382355 (+ 68us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 11:46:20.382356 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 11:46:20.382381 (+ 25us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 11:46:20.382432 (+ 51us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 11:46:20.382634 (+ 202us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 11:46:20.383134 (+ 500us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 11:46:20.384210 (+ 1076us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 11:46:20.384575 (+ 365us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 11:46:20.384667 (+ 92us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 11:46:20.384674 (+ 7us) server_negotiation.cc:1035] Waiting for connection context
0620 11:46:20.384848 (+ 174us) server_negotiation.cc:299] Negotiation successful
0620 11:46:20.384887 (+ 39us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":24}
I20260620 11:46:20.385181 26682 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 11:46:20.382038 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.234.194:38075 (local address 127.6.234.193:47379)
0620 11:46:20.382183 (+ 145us) negotiation.cc:107] Waiting for socket to connect
0620 11:46:20.382195 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0620 11:46:20.382280 (+ 85us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 11:46:20.382450 (+ 170us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 11:46:20.382452 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 11:46:20.382454 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 11:46:20.382539 (+ 85us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 11:46:20.382542 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 11:46:20.383217 (+ 675us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 11:46:20.383218 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 11:46:20.384087 (+ 869us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 11:46:20.384089 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 11:46:20.384689 (+ 600us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 11:46:20.384690 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 11:46:20.384762 (+ 72us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 11:46:20.384767 (+ 5us) client_negotiation.cc:769] Sending connection context
0620 11:46:20.384826 (+ 59us) client_negotiation.cc:240] Negotiation successful
0620 11:46:20.384864 (+ 38us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":123,"thread_start_us":73,"threads_started":1}
I20260620 11:46:20.385366 26610 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "61ce3f703e8d4ece9d1adce07d3f7ad5" candidate_uuid: "180bb34b014448ffa2daa5f7698fa698" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0142ad50ae1a4d429b744b5731e6a340" is_pre_election: true
I20260620 11:46:20.385543 26610 raft_consensus.cc:2454] T 61ce3f703e8d4ece9d1adce07d3f7ad5 P 0142ad50ae1a4d429b744b5731e6a340 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 180bb34b014448ffa2daa5f7698fa698 in term 0.
I20260620 11:46:20.385545 26476 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "61ce3f703e8d4ece9d1adce07d3f7ad5" candidate_uuid: "180bb34b014448ffa2daa5f7698fa698" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "495d7e3367924702b1ce9ba3ce523b6e" is_pre_election: true
I20260620 11:46:20.385741 26476 raft_consensus.cc:2454] T 61ce3f703e8d4ece9d1adce07d3f7ad5 P 495d7e3367924702b1ce9ba3ce523b6e [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 180bb34b014448ffa2daa5f7698fa698 in term 0.
I20260620 11:46:20.385813 26278 leader_election.cc:302] T 61ce3f703e8d4ece9d1adce07d3f7ad5 P 180bb34b014448ffa2daa5f7698fa698 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0142ad50ae1a4d429b744b5731e6a340, 180bb34b014448ffa2daa5f7698fa698; no voters:
I20260620 11:46:20.385959 26679 raft_consensus.cc:2781] T 61ce3f703e8d4ece9d1adce07d3f7ad5 P 180bb34b014448ffa2daa5f7698fa698 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260620 11:46:20.386016 26679 raft_consensus.cc:482] T 61ce3f703e8d4ece9d1adce07d3f7ad5 P 180bb34b014448ffa2daa5f7698fa698 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260620 11:46:20.386042 26679 raft_consensus.cc:3037] T 61ce3f703e8d4ece9d1adce07d3f7ad5 P 180bb34b014448ffa2daa5f7698fa698 [term 0 FOLLOWER]: Advancing to term 1
I20260620 11:46:20.386734 26679 raft_consensus.cc:504] T 61ce3f703e8d4ece9d1adce07d3f7ad5 P 180bb34b014448ffa2daa5f7698fa698 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "495d7e3367924702b1ce9ba3ce523b6e" member_type: VOTER last_known_addr { host: "127.6.234.194" port: 38075 } } peers { permanent_uuid: "0142ad50ae1a4d429b744b5731e6a340" member_type: VOTER last_known_addr { host: "127.6.234.195" port: 44799 } } peers { permanent_uuid: "180bb34b014448ffa2daa5f7698fa698" member_type: VOTER last_known_addr { host: "127.6.234.193" port: 46439 } }
I20260620 11:46:20.386876 26679 leader_election.cc:288] T 61ce3f703e8d4ece9d1adce07d3f7ad5 P 180bb34b014448ffa2daa5f7698fa698 [CANDIDATE]: Term 1 election: Requested vote from peers 495d7e3367924702b1ce9ba3ce523b6e (127.6.234.194:38075), 0142ad50ae1a4d429b744b5731e6a340 (127.6.234.195:44799)
I20260620 11:46:20.387012 26476 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "61ce3f703e8d4ece9d1adce07d3f7ad5" candidate_uuid: "180bb34b014448ffa2daa5f7698fa698" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "495d7e3367924702b1ce9ba3ce523b6e"
I20260620 11:46:20.387081 26476 raft_consensus.cc:3037] T 61ce3f703e8d4ece9d1adce07d3f7ad5 P 495d7e3367924702b1ce9ba3ce523b6e [term 0 FOLLOWER]: Advancing to term 1
I20260620 11:46:20.387781 26476 raft_consensus.cc:2454] T 61ce3f703e8d4ece9d1adce07d3f7ad5 P 495d7e3367924702b1ce9ba3ce523b6e [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 180bb34b014448ffa2daa5f7698fa698 in term 1.
I20260620 11:46:20.387903 26610 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "61ce3f703e8d4ece9d1adce07d3f7ad5" candidate_uuid: "180bb34b014448ffa2daa5f7698fa698" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0142ad50ae1a4d429b744b5731e6a340"
I20260620 11:46:20.387964 26275 leader_election.cc:302] T 61ce3f703e8d4ece9d1adce07d3f7ad5 P 180bb34b014448ffa2daa5f7698fa698 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 180bb34b014448ffa2daa5f7698fa698, 495d7e3367924702b1ce9ba3ce523b6e; no voters:
I20260620 11:46:20.387990 26610 raft_consensus.cc:3037] T 61ce3f703e8d4ece9d1adce07d3f7ad5 P 0142ad50ae1a4d429b744b5731e6a340 [term 0 FOLLOWER]: Advancing to term 1
I20260620 11:46:20.388063 26679 raft_consensus.cc:2781] T 61ce3f703e8d4ece9d1adce07d3f7ad5 P 180bb34b014448ffa2daa5f7698fa698 [term 1 FOLLOWER]: Leader election won for term 1
I20260620 11:46:20.388229 26679 raft_consensus.cc:686] T 61ce3f703e8d4ece9d1adce07d3f7ad5 P 180bb34b014448ffa2daa5f7698fa698 [term 1 LEADER]: Becoming Leader. State: Replica: 180bb34b014448ffa2daa5f7698fa698, State: Running, Role: LEADER
I20260620 11:46:20.388329 26679 consensus_queue.cc:237] T 61ce3f703e8d4ece9d1adce07d3f7ad5 P 180bb34b014448ffa2daa5f7698fa698 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "495d7e3367924702b1ce9ba3ce523b6e" member_type: VOTER last_known_addr { host: "127.6.234.194" port: 38075 } } peers { permanent_uuid: "0142ad50ae1a4d429b744b5731e6a340" member_type: VOTER last_known_addr { host: "127.6.234.195" port: 44799 } } peers { permanent_uuid: "180bb34b014448ffa2daa5f7698fa698" member_type: VOTER last_known_addr { host: "127.6.234.193" port: 46439 } }
I20260620 11:46:20.388899 26610 raft_consensus.cc:2454] T 61ce3f703e8d4ece9d1adce07d3f7ad5 P 0142ad50ae1a4d429b744b5731e6a340 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 180bb34b014448ffa2daa5f7698fa698 in term 1.
I20260620 11:46:20.389218 26197 catalog_manager.cc:5495] T 61ce3f703e8d4ece9d1adce07d3f7ad5 P 180bb34b014448ffa2daa5f7698fa698 reported cstate change: term changed from 0 to 1, leader changed from <none> to 180bb34b014448ffa2daa5f7698fa698 (127.6.234.193). New cstate: current_term: 1 leader_uuid: "180bb34b014448ffa2daa5f7698fa698" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "495d7e3367924702b1ce9ba3ce523b6e" member_type: VOTER last_known_addr { host: "127.6.234.194" port: 38075 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "0142ad50ae1a4d429b744b5731e6a340" member_type: VOTER last_known_addr { host: "127.6.234.195" port: 44799 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "180bb34b014448ffa2daa5f7698fa698" member_type: VOTER last_known_addr { host: "127.6.234.193" port: 46439 } health_report { overall_health: HEALTHY } } }
I20260620 11:46:20.407675 26672 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 11:46:20.405232 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48076 (local address 127.6.234.193:46439)
0620 11:46:20.405297 (+ 65us) server_negotiation.cc:206] Beginning negotiation
0620 11:46:20.405299 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 11:46:20.405319 (+ 20us) server_negotiation.cc:407] Connection header received
0620 11:46:20.405362 (+ 43us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 11:46:20.405362 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 11:46:20.405379 (+ 17us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 11:46:20.405453 (+ 74us) server_negotiation.cc:226] Negotiated authn=TOKEN
0620 11:46:20.405605 (+ 152us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 11:46:20.406097 (+ 492us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 11:46:20.406870 (+ 773us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 11:46:20.407070 (+ 200us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 11:46:20.407148 (+ 78us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 11:46:20.407381 (+ 233us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0620 11:46:20.407484 (+ 103us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0620 11:46:20.407566 (+ 82us) server_negotiation.cc:1035] Waiting for connection context
0620 11:46:20.407600 (+ 34us) server_negotiation.cc:299] Negotiation successful
0620 11:46:20.407619 (+ 19us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":40}
I20260620 11:46:20.409678 26610 raft_consensus.cc:1261] T 61ce3f703e8d4ece9d1adce07d3f7ad5 P 0142ad50ae1a4d429b744b5731e6a340 [term 1 FOLLOWER]: Refusing update from remote peer 180bb34b014448ffa2daa5f7698fa698: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 11:46:20.409680 26476 raft_consensus.cc:1261] T 61ce3f703e8d4ece9d1adce07d3f7ad5 P 495d7e3367924702b1ce9ba3ce523b6e [term 1 FOLLOWER]: Refusing update from remote peer 180bb34b014448ffa2daa5f7698fa698: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 11:46:20.410094 26684 consensus_queue.cc:1035] T 61ce3f703e8d4ece9d1adce07d3f7ad5 P 180bb34b014448ffa2daa5f7698fa698 [LEADER]: Connected to new peer: Peer: permanent_uuid: "495d7e3367924702b1ce9ba3ce523b6e" member_type: VOTER last_known_addr { host: "127.6.234.194" port: 38075 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 11:46:20.410218 26679 consensus_queue.cc:1035] T 61ce3f703e8d4ece9d1adce07d3f7ad5 P 180bb34b014448ffa2daa5f7698fa698 [LEADER]: Connected to new peer: Peer: permanent_uuid: "0142ad50ae1a4d429b744b5731e6a340" member_type: VOTER last_known_addr { host: "127.6.234.195" port: 44799 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 11:46:20.412582 26689 mvcc.cc:205] Tried to move back new op lower bound from 7298891695754813440 to 7298891695671123968. Current Snapshot: MvccSnapshot[applied={T|T < 7298891695754813440}]
I20260620 11:46:20.412976 26687 mvcc.cc:205] Tried to move back new op lower bound from 7298891695754813440 to 7298891695671123968. Current Snapshot: MvccSnapshot[applied={T|T < 7298891695754813440}]
I20260620 11:46:20.414951 26197 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:41538:
table { table_name: "test-table" } modify_external_catalogs: true
I20260620 11:46:20.415055 26197 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:41538:
table { table_name: "test-table" } modify_external_catalogs: true
I20260620 11:46:20.415817 26197 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 6150208d5e044cc9b5e38215bc21010c: Sending DeleteTablet for 3 replicas of tablet 61ce3f703e8d4ece9d1adce07d3f7ad5
I20260620 11:46:20.416157 26590 tablet_service.cc:1514] Processing DeleteTablet for tablet 61ce3f703e8d4ece9d1adce07d3f7ad5 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 11:46:20 UTC) from {username='kudu', principal='kudu/127.6.234.254@KRBTEST.COM'} at 127.0.0.1:46162
I20260620 11:46:20.416227 26456 tablet_service.cc:1514] Processing DeleteTablet for tablet 61ce3f703e8d4ece9d1adce07d3f7ad5 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 11:46:20 UTC) from {username='kudu', principal='kudu/127.6.234.254@KRBTEST.COM'} at 127.0.0.1:51466
I20260620 11:46:20.416275 26322 tablet_service.cc:1514] Processing DeleteTablet for tablet 61ce3f703e8d4ece9d1adce07d3f7ad5 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 11:46:20 UTC) from {username='kudu', principal='kudu/127.6.234.254@KRBTEST.COM'} at 127.0.0.1:48060
I20260620 11:46:20.416361 26696 tablet_replica.cc:330] T 61ce3f703e8d4ece9d1adce07d3f7ad5 P 0142ad50ae1a4d429b744b5731e6a340: stopping tablet replica
I20260620 11:46:20.416451 26696 raft_consensus.cc:2229] T 61ce3f703e8d4ece9d1adce07d3f7ad5 P 0142ad50ae1a4d429b744b5731e6a340 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260620 11:46:20.416463 26698 tablet_replica.cc:330] T 61ce3f703e8d4ece9d1adce07d3f7ad5 P 180bb34b014448ffa2daa5f7698fa698: stopping tablet replica
I20260620 11:46:20.416548 26698 raft_consensus.cc:2229] T 61ce3f703e8d4ece9d1adce07d3f7ad5 P 180bb34b014448ffa2daa5f7698fa698 [term 1 LEADER]: Raft consensus shutting down.
I20260620 11:46:20.416559 26696 raft_consensus.cc:2258] T 61ce3f703e8d4ece9d1adce07d3f7ad5 P 0142ad50ae1a4d429b744b5731e6a340 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 11:46:20.416654 26698 raft_consensus.cc:2258] T 61ce3f703e8d4ece9d1adce07d3f7ad5 P 180bb34b014448ffa2daa5f7698fa698 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 11:46:20.417016 26696 ts_tablet_manager.cc:1905] T 61ce3f703e8d4ece9d1adce07d3f7ad5 P 0142ad50ae1a4d429b744b5731e6a340: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 11:46:20.417074 7083 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task1JRg7N/build/release/bin/kudu with pid 26259
I20260620 11:46:20.417078 26698 ts_tablet_manager.cc:1905] T 61ce3f703e8d4ece9d1adce07d3f7ad5 P 180bb34b014448ffa2daa5f7698fa698: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 11:46:20.418848 26696 ts_tablet_manager.cc:1918] T 61ce3f703e8d4ece9d1adce07d3f7ad5 P 0142ad50ae1a4d429b744b5731e6a340: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260620 11:46:20.418913 26696 log.cc:1196] T 61ce3f703e8d4ece9d1adce07d3f7ad5 P 0142ad50ae1a4d429b744b5731e6a340: Deleting WAL directory at /tmp/dist-test-task1JRg7N/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781955746769051-7083-0/minicluster-data/ts-2/wal/wals/61ce3f703e8d4ece9d1adce07d3f7ad5
I20260620 11:46:20.419183 26696 ts_tablet_manager.cc:1939] T 61ce3f703e8d4ece9d1adce07d3f7ad5 P 0142ad50ae1a4d429b744b5731e6a340: Deleting consensus metadata
I20260620 11:46:20.419585 26184 catalog_manager.cc:4841] TS 0142ad50ae1a4d429b744b5731e6a340 (127.6.234.195:44799): tablet 61ce3f703e8d4ece9d1adce07d3f7ad5 (table test-table [id=9b299d84348d43daaa66453804ddd4a3]) successfully deleted
I20260620 11:46:20.419817 26697 tablet_replica.cc:330] T 61ce3f703e8d4ece9d1adce07d3f7ad5 P 495d7e3367924702b1ce9ba3ce523b6e: stopping tablet replica
I20260620 11:46:20.419937 26697 raft_consensus.cc:2229] T 61ce3f703e8d4ece9d1adce07d3f7ad5 P 495d7e3367924702b1ce9ba3ce523b6e [term 1 FOLLOWER]: Raft consensus shutting down.
I20260620 11:46:20.420066 26697 raft_consensus.cc:2258] T 61ce3f703e8d4ece9d1adce07d3f7ad5 P 495d7e3367924702b1ce9ba3ce523b6e [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 11:46:20.420431 26697 ts_tablet_manager.cc:1905] T 61ce3f703e8d4ece9d1adce07d3f7ad5 P 495d7e3367924702b1ce9ba3ce523b6e: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 11:46:20.421820 26697 ts_tablet_manager.cc:1918] T 61ce3f703e8d4ece9d1adce07d3f7ad5 P 495d7e3367924702b1ce9ba3ce523b6e: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260620 11:46:20.421905 26697 log.cc:1196] T 61ce3f703e8d4ece9d1adce07d3f7ad5 P 495d7e3367924702b1ce9ba3ce523b6e: Deleting WAL directory at /tmp/dist-test-task1JRg7N/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781955746769051-7083-0/minicluster-data/ts-1/wal/wals/61ce3f703e8d4ece9d1adce07d3f7ad5
I20260620 11:46:20.422151 26697 ts_tablet_manager.cc:1939] T 61ce3f703e8d4ece9d1adce07d3f7ad5 P 495d7e3367924702b1ce9ba3ce523b6e: Deleting consensus metadata
I20260620 11:46:20.422537 26181 catalog_manager.cc:4841] TS 495d7e3367924702b1ce9ba3ce523b6e (127.6.234.194:38075): tablet 61ce3f703e8d4ece9d1adce07d3f7ad5 (table test-table [id=9b299d84348d43daaa66453804ddd4a3]) successfully deleted
W20260620 11:46:20.423592 26182 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.6.234.193:46439 (error 108)
I20260620 11:46:20.423838 7083 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task1JRg7N/build/release/bin/kudu with pid 26393
I20260620 11:46:20.424598 26669 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 11:46:20.423996 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.234.193:46439 (local address 127.0.0.1:48080)
0620 11:46:20.424514 (+ 518us) negotiation.cc:107] Waiting for socket to connect
0620 11:46:20.424557 (+ 43us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.234.193:46439: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":498}
W20260620 11:46:20.424994 26182 catalog_manager.cc:4584] TS 180bb34b014448ffa2daa5f7698fa698 (127.6.234.193:46439): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 61ce3f703e8d4ece9d1adce07d3f7ad5: Network error: Client connection negotiation failed: client connection to 127.6.234.193:46439: connect: Connection refused (error 111)
I20260620 11:46:20.428952 7083 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task1JRg7N/build/release/bin/kudu with pid 26527
I20260620 11:46:20.433650 7083 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task1JRg7N/build/release/bin/kudu with pid 26165
2026-06-20T11:46:20Z chronyd exiting
[ OK ] EncryptionPolicyTest.TestEncryptionPolicy/4 (2063 ms)
[ RUN ] EncryptionPolicyTest.TestEncryptionPolicy/5
2026-06-20T11:46:20Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-20T11:46:20Z Disabled control of system clock
I20260620 11:46:20.460637 7083 external_mini_cluster.cc:1326] Running /tmp/dist-test-task1JRg7N/build/release/bin/kudu
/tmp/dist-test-task1JRg7N/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task1JRg7N/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781955746769051-7083-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task1JRg7N/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781955746769051-7083-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task1JRg7N/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781955746769051-7083-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task1JRg7N/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781955746769051-7083-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.6.234.254:36373
--webserver_interface=127.6.234.254
--webserver_port=0
--builtin_ntp_servers=127.6.234.212:44425
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.6.234.254:36373
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
W20260620 11:46:20.540324 26703 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 11:46:20.540539 26703 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 11:46:20.540560 26703 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 11:46:20.541638 26703 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 11:46:20.541671 26703 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 11:46:20.541684 26703 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 11:46:20.541720 26703 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 11:46:20.541733 26703 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 11:46:20.542997 26703 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.6.234.212:44425
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task1JRg7N/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781955746769051-7083-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task1JRg7N/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781955746769051-7083-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.6.234.254:36373
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.6.234.254:36373
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task1JRg7N/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781955746769051-7083-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.6.234.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.26703
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task1JRg7N/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781955746769051-7083-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 20 Jun 2026 10:32:37 UTC on ccycloud.quasar-bgyofv.root.comops.site
build id 4457173
I20260620 11:46:20.543192 26703 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 11:46:20.543421 26703 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 11:46:20.548914 26708 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 11:46:20.548981 26709 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 11:46:20.548893 26711 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 11:46:20.549165 26703 server_base.cc:1034] running on GCE node
I20260620 11:46:20.549399 26703 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 11:46:20.549633 26703 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 11:46:20.550771 26703 hybrid_clock.cc:648] HybridClock initialized: now 1781955980550760 us; error 29 us; skew 500 ppm
I20260620 11:46:20.555460 26703 webserver.cc:460] Webserver started at http://127.6.234.254:35399/ using document root <none> and password file <none>
I20260620 11:46:20.555727 26703 fs_manager.cc:359] Metadata directory not provided
I20260620 11:46:20.555776 26703 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 11:46:20.555933 26703 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 11:46:20.557049 26703 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task1JRg7N/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781955746769051-7083-0/minicluster-data/master-0/data/instance:
uuid: "17d389e1a8b0416db7b1743fc8696131"
format_stamp: "Formatted at 2026-06-20 11:46:20 on dist-test-slave-cjc6"
server_key: "db6ee974d1e15701a08b39b264a0c127"
server_key_iv: "445e83bf859380e328350a472dfba1f8"
server_key_version: "encryptionkey@0"
I20260620 11:46:20.557416 26703 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task1JRg7N/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781955746769051-7083-0/minicluster-data/master-0/wal/instance:
uuid: "17d389e1a8b0416db7b1743fc8696131"
format_stamp: "Formatted at 2026-06-20 11:46:20 on dist-test-slave-cjc6"
server_key: "db6ee974d1e15701a08b39b264a0c127"
server_key_iv: "445e83bf859380e328350a472dfba1f8"
server_key_version: "encryptionkey@0"
I20260620 11:46:20.558782 26703 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260620 11:46:20.559518 26717 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 11:46:20.559700 26703 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260620 11:46:20.559800 26703 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task1JRg7N/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781955746769051-7083-0/minicluster-data/master-0/data,/tmp/dist-test-task1JRg7N/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781955746769051-7083-0/minicluster-data/master-0/wal
uuid: "17d389e1a8b0416db7b1743fc8696131"
format_stamp: "Formatted at 2026-06-20 11:46:20 on dist-test-slave-cjc6"
server_key: "db6ee974d1e15701a08b39b264a0c127"
server_key_iv: "445e83bf859380e328350a472dfba1f8"
server_key_version: "encryptionkey@0"
I20260620 11:46:20.559886 26703 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task1JRg7N/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781955746769051-7083-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task1JRg7N/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781955746769051-7083-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task1JRg7N/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781955746769051-7083-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 11:46:20.577104 26703 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 11:46:20.577431 26703 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 11:46:20.577591 26703 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 11:46:20.581498 26703 rpc_server.cc:307] RPC server started. Bound to: 127.6.234.254:36373
I20260620 11:46:20.581534 26769 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.6.234.254:36373 every 8 connection(s)
I20260620 11:46:20.581910 26703 server_base.cc:1166] Dumped server information to /tmp/dist-test-task1JRg7N/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781955746769051-7083-0/minicluster-data/master-0/data/info.pb
I20260620 11:46:20.582873 26770 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 11:46:20.584937 7083 external_mini_cluster.cc:1388] Started /tmp/dist-test-task1JRg7N/build/release/bin/kudu as pid 26703
I20260620 11:46:20.585047 7083 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task1JRg7N/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781955746769051-7083-0/minicluster-data/master-0/wal/instance
I20260620 11:46:20.585261 7083 external_mini_cluster.cc:1428] Setting key f144c35efbcb7d2b8aa113984e8aeb0d
I20260620 11:46:20.585352 26770 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 17d389e1a8b0416db7b1743fc8696131: Bootstrap starting.
I20260620 11:46:20.586099 26770 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 17d389e1a8b0416db7b1743fc8696131: Neither blocks nor log segments found. Creating new log.
I20260620 11:46:20.586405 26770 log.cc:824] T 00000000000000000000000000000000 P 17d389e1a8b0416db7b1743fc8696131: Log is configured to *not* fsync() on all Append() calls
I20260620 11:46:20.587056 26770 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 17d389e1a8b0416db7b1743fc8696131: No bootstrap required, opened a new log
I20260620 11:46:20.588784 26770 raft_consensus.cc:348] T 00000000000000000000000000000000 P 17d389e1a8b0416db7b1743fc8696131 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "17d389e1a8b0416db7b1743fc8696131" member_type: VOTER last_known_addr { host: "127.6.234.254" port: 36373 } }
I20260620 11:46:20.588914 26770 raft_consensus.cc:374] T 00000000000000000000000000000000 P 17d389e1a8b0416db7b1743fc8696131 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 11:46:20.588935 26770 raft_consensus.cc:729] T 00000000000000000000000000000000 P 17d389e1a8b0416db7b1743fc8696131 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 17d389e1a8b0416db7b1743fc8696131, State: Initialized, Role: FOLLOWER
I20260620 11:46:20.589073 26770 consensus_queue.cc:260] T 00000000000000000000000000000000 P 17d389e1a8b0416db7b1743fc8696131 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "17d389e1a8b0416db7b1743fc8696131" member_type: VOTER last_known_addr { host: "127.6.234.254" port: 36373 } }
I20260620 11:46:20.589143 26770 raft_consensus.cc:388] T 00000000000000000000000000000000 P 17d389e1a8b0416db7b1743fc8696131 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260620 11:46:20.589184 26770 raft_consensus.cc:482] T 00000000000000000000000000000000 P 17d389e1a8b0416db7b1743fc8696131 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260620 11:46:20.589229 26770 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 17d389e1a8b0416db7b1743fc8696131 [term 0 FOLLOWER]: Advancing to term 1
I20260620 11:46:20.589818 26770 raft_consensus.cc:504] T 00000000000000000000000000000000 P 17d389e1a8b0416db7b1743fc8696131 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "17d389e1a8b0416db7b1743fc8696131" member_type: VOTER last_known_addr { host: "127.6.234.254" port: 36373 } }
I20260620 11:46:20.589926 26770 leader_election.cc:302] T 00000000000000000000000000000000 P 17d389e1a8b0416db7b1743fc8696131 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 17d389e1a8b0416db7b1743fc8696131; no voters:
I20260620 11:46:20.590099 26770 leader_election.cc:288] T 00000000000000000000000000000000 P 17d389e1a8b0416db7b1743fc8696131 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260620 11:46:20.590142 26775 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 17d389e1a8b0416db7b1743fc8696131 [term 1 FOLLOWER]: Leader election won for term 1
I20260620 11:46:20.590277 26775 raft_consensus.cc:686] T 00000000000000000000000000000000 P 17d389e1a8b0416db7b1743fc8696131 [term 1 LEADER]: Becoming Leader. State: Replica: 17d389e1a8b0416db7b1743fc8696131, State: Running, Role: LEADER
I20260620 11:46:20.590369 26775 consensus_queue.cc:237] T 00000000000000000000000000000000 P 17d389e1a8b0416db7b1743fc8696131 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "17d389e1a8b0416db7b1743fc8696131" member_type: VOTER last_known_addr { host: "127.6.234.254" port: 36373 } }
I20260620 11:46:20.590421 26770 sys_catalog.cc:564] T 00000000000000000000000000000000 P 17d389e1a8b0416db7b1743fc8696131 [sys.catalog]: configured and running, proceeding with master startup.
I20260620 11:46:20.590706 26775 sys_catalog.cc:455] T 00000000000000000000000000000000 P 17d389e1a8b0416db7b1743fc8696131 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 17d389e1a8b0416db7b1743fc8696131. Latest consensus state: current_term: 1 leader_uuid: "17d389e1a8b0416db7b1743fc8696131" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "17d389e1a8b0416db7b1743fc8696131" member_type: VOTER last_known_addr { host: "127.6.234.254" port: 36373 } } }
I20260620 11:46:20.590713 26776 sys_catalog.cc:455] T 00000000000000000000000000000000 P 17d389e1a8b0416db7b1743fc8696131 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "17d389e1a8b0416db7b1743fc8696131" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "17d389e1a8b0416db7b1743fc8696131" member_type: VOTER last_known_addr { host: "127.6.234.254" port: 36373 } } }
I20260620 11:46:20.590797 26775 sys_catalog.cc:458] T 00000000000000000000000000000000 P 17d389e1a8b0416db7b1743fc8696131 [sys.catalog]: This master's current role is: LEADER
I20260620 11:46:20.590806 26776 sys_catalog.cc:458] T 00000000000000000000000000000000 P 17d389e1a8b0416db7b1743fc8696131 [sys.catalog]: This master's current role is: LEADER
I20260620 11:46:20.591168 26773 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 11:46:20.585829 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38818 (local address 127.6.234.254:36373)
0620 11:46:20.586034 (+ 205us) server_negotiation.cc:206] Beginning negotiation
0620 11:46:20.586041 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0620 11:46:20.586066 (+ 25us) server_negotiation.cc:407] Connection header received
0620 11:46:20.586566 (+ 500us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 11:46:20.586573 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 11:46:20.586633 (+ 60us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 11:46:20.586778 (+ 145us) server_negotiation.cc:226] Negotiated authn=SASL
0620 11:46:20.587018 (+ 240us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 11:46:20.587925 (+ 907us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 11:46:20.588749 (+ 824us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 11:46:20.589335 (+ 586us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 11:46:20.589465 (+ 130us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 11:46:20.590240 (+ 775us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 11:46:20.590244 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 11:46:20.590244 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 11:46:20.590450 (+ 206us) server_negotiation.cc:922] calling sasl_server_start()
0620 11:46:20.590505 (+ 55us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 11:46:20.590520 (+ 15us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 11:46:20.590582 (+ 62us) server_negotiation.cc:1035] Waiting for connection context
0620 11:46:20.590867 (+ 285us) server_negotiation.cc:299] Negotiation successful
0620 11:46:20.590955 (+ 88us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":114,"thread_start_us":85,"threads_started":1}
I20260620 11:46:20.591749 26780 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260620 11:46:20.592345 26780 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260620 11:46:20.593729 26780 catalog_manager.cc:1347] Generated new cluster ID: b9213e385a64402f86acf24f529ff2cc
I20260620 11:46:20.593782 26780 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260620 11:46:20.613140 26780 catalog_manager.cc:1370] Generated new certificate authority record
I20260620 11:46:20.613724 26780 catalog_manager.cc:1504] Loading token signing keys...
I20260620 11:46:20.618649 26780 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 17d389e1a8b0416db7b1743fc8696131: Generated new TSK 0
I20260620 11:46:20.618824 26780 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260620 11:46:20.622315 7083 external_mini_cluster.cc:1326] Running /tmp/dist-test-task1JRg7N/build/release/bin/kudu
/tmp/dist-test-task1JRg7N/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task1JRg7N/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781955746769051-7083-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-task1JRg7N/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781955746769051-7083-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task1JRg7N/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781955746769051-7083-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-task1JRg7N/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781955746769051-7083-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.6.234.193:0
--local_ip_for_outbound_sockets=127.6.234.193
--webserver_interface=127.6.234.193
--webserver_port=0
--tserver_master_addrs=127.6.234.254:36373
--builtin_ntp_servers=127.6.234.212:44425
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
W20260620 11:46:20.702890 26794 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 11:46:20.703126 26794 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 11:46:20.703161 26794 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 11:46:20.704293 26794 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 11:46:20.704355 26794 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 11:46:20.704396 26794 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.6.234.193
I20260620 11:46:20.705783 26794 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.6.234.212:44425
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task1JRg7N/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781955746769051-7083-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-task1JRg7N/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781955746769051-7083-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.6.234.193:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task1JRg7N/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781955746769051-7083-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.6.234.193
--webserver_port=0
--tserver_master_addrs=127.6.234.254:36373
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.26794
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.6.234.193
--log_dir=/tmp/dist-test-task1JRg7N/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781955746769051-7083-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 20 Jun 2026 10:32:37 UTC on ccycloud.quasar-bgyofv.root.comops.site
build id 4457173
I20260620 11:46:20.706045 26794 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 11:46:20.706342 26794 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 11:46:20.711957 26802 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 11:46:20.711964 26799 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 11:46:20.712049 26794 server_base.cc:1034] running on GCE node
W20260620 11:46:20.711957 26800 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 11:46:20.712497 26794 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 11:46:20.712774 26794 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 11:46:20.713943 26794 hybrid_clock.cc:648] HybridClock initialized: now 1781955980713926 us; error 31 us; skew 500 ppm
I20260620 11:46:20.715283 26794 webserver.cc:460] Webserver started at http://127.6.234.193:38239/ using document root <none> and password file <none>
I20260620 11:46:20.715559 26794 fs_manager.cc:359] Metadata directory not provided
I20260620 11:46:20.715623 26794 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 11:46:20.715754 26794 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 11:46:20.716909 26794 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task1JRg7N/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781955746769051-7083-0/minicluster-data/ts-0/data/instance:
uuid: "f6d343cd73fd43b7a7c7f2bc124b0fa9"
format_stamp: "Formatted at 2026-06-20 11:46:20 on dist-test-slave-cjc6"
server_key: "65d7038cf506380092a9b36297883c7f"
server_key_iv: "2b1752d193174fa453c91877a353d7fe"
server_key_version: "encryptionkey@0"
I20260620 11:46:20.717202 26794 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task1JRg7N/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781955746769051-7083-0/minicluster-data/ts-0/wal/instance:
uuid: "f6d343cd73fd43b7a7c7f2bc124b0fa9"
format_stamp: "Formatted at 2026-06-20 11:46:20 on dist-test-slave-cjc6"
server_key: "65d7038cf506380092a9b36297883c7f"
server_key_iv: "2b1752d193174fa453c91877a353d7fe"
server_key_version: "encryptionkey@0"
I20260620 11:46:20.718453 26794 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260620 11:46:20.719076 26808 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 11:46:20.719238 26794 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260620 11:46:20.719334 26794 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task1JRg7N/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781955746769051-7083-0/minicluster-data/ts-0/data,/tmp/dist-test-task1JRg7N/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781955746769051-7083-0/minicluster-data/ts-0/wal
uuid: "f6d343cd73fd43b7a7c7f2bc124b0fa9"
format_stamp: "Formatted at 2026-06-20 11:46:20 on dist-test-slave-cjc6"
server_key: "65d7038cf506380092a9b36297883c7f"
server_key_iv: "2b1752d193174fa453c91877a353d7fe"
server_key_version: "encryptionkey@0"
I20260620 11:46:20.719424 26794 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task1JRg7N/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781955746769051-7083-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-task1JRg7N/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781955746769051-7083-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-task1JRg7N/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781955746769051-7083-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 11:46:20.732343 26794 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 11:46:20.732635 26794 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 11:46:20.732784 26794 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 11:46:20.733013 26794 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 11:46:20.733318 26794 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 11:46:20.733372 26794 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 11:46:20.733412 26794 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 11:46:20.733436 26794 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 11:46:20.739218 26794 rpc_server.cc:307] RPC server started. Bound to: 127.6.234.193:34367
I20260620 11:46:20.739267 26921 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.6.234.193:34367 every 8 connection(s)
I20260620 11:46:20.739573 26794 server_base.cc:1166] Dumped server information to /tmp/dist-test-task1JRg7N/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781955746769051-7083-0/minicluster-data/ts-0/data/info.pb
I20260620 11:46:20.744318 26773 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 11:46:20.740220 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.6.234.193:33691 (local address 127.6.234.254:36373)
0620 11:46:20.740265 (+ 45us) server_negotiation.cc:206] Beginning negotiation
0620 11:46:20.740267 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 11:46:20.740507 (+ 240us) server_negotiation.cc:407] Connection header received
0620 11:46:20.740965 (+ 458us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 11:46:20.740966 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 11:46:20.740975 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 11:46:20.741035 (+ 60us) server_negotiation.cc:226] Negotiated authn=SASL
0620 11:46:20.741676 (+ 641us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 11:46:20.742219 (+ 543us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 11:46:20.743140 (+ 921us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 11:46:20.743314 (+ 174us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 11:46:20.743362 (+ 48us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 11:46:20.744099 (+ 737us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 11:46:20.744100 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 11:46:20.744101 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 11:46:20.744105 (+ 4us) server_negotiation.cc:922] calling sasl_server_start()
0620 11:46:20.744118 (+ 13us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 11:46:20.744121 (+ 3us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 11:46:20.744164 (+ 43us) server_negotiation.cc:1035] Waiting for connection context
0620 11:46:20.744244 (+ 80us) server_negotiation.cc:299] Negotiation successful
0620 11:46:20.744262 (+ 18us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":19}
I20260620 11:46:20.744513 26924 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 11:46:20.740187 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.234.254:36373 (local address 127.6.234.193:33691)
0620 11:46:20.740369 (+ 182us) negotiation.cc:107] Waiting for socket to connect
0620 11:46:20.740409 (+ 40us) client_negotiation.cc:174] Beginning negotiation
0620 11:46:20.740884 (+ 475us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 11:46:20.741111 (+ 227us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 11:46:20.741114 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 11:46:20.741122 (+ 8us) client_negotiation.cc:189] Negotiated authn=SASL
0620 11:46:20.741597 (+ 475us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 11:46:20.741601 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 11:46:20.742298 (+ 697us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 11:46:20.742298 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 11:46:20.743065 (+ 767us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 11:46:20.743066 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 11:46:20.743385 (+ 319us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 11:46:20.743386 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 11:46:20.743488 (+ 102us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 11:46:20.743963 (+ 475us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0620 11:46:20.743964 (+ 1us) client_negotiation.cc:656] calling sasl_client_start()
0620 11:46:20.743997 (+ 33us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0620 11:46:20.744018 (+ 21us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 11:46:20.744175 (+ 157us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 11:46:20.744178 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 11:46:20.744180 (+ 2us) client_negotiation.cc:769] Sending connection context
0620 11:46:20.744231 (+ 51us) client_negotiation.cc:240] Negotiation successful
0620 11:46:20.744303 (+ 72us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":120,"thread_start_us":87,"threads_started":1}
I20260620 11:46:20.745119 26922 heartbeater.cc:347] Connected to a master server at 127.6.234.254:36373
I20260620 11:46:20.745208 26922 heartbeater.cc:464] Registering TS with master...
I20260620 11:46:20.745362 26922 heartbeater.cc:511] Master 127.6.234.254:36373 requested a full tablet report, sending...
I20260620 11:46:20.745939 26734 ts_manager.cc:194] Registered new tserver with Master: f6d343cd73fd43b7a7c7f2bc124b0fa9 (127.6.234.193:34367)
I20260620 11:46:20.746501 26734 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.6.234.193:33691
I20260620 11:46:20.747346 7083 external_mini_cluster.cc:1388] Started /tmp/dist-test-task1JRg7N/build/release/bin/kudu as pid 26794
I20260620 11:46:20.747429 7083 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task1JRg7N/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781955746769051-7083-0/minicluster-data/ts-0/wal/instance
I20260620 11:46:20.747659 7083 external_mini_cluster.cc:1428] Setting key 4ffd29a6df2c122ab8839948bda21655
I20260620 11:46:20.748518 7083 external_mini_cluster.cc:1326] Running /tmp/dist-test-task1JRg7N/build/release/bin/kudu
/tmp/dist-test-task1JRg7N/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task1JRg7N/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781955746769051-7083-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-task1JRg7N/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781955746769051-7083-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task1JRg7N/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781955746769051-7083-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-task1JRg7N/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781955746769051-7083-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.6.234.194:0
--local_ip_for_outbound_sockets=127.6.234.194
--webserver_interface=127.6.234.194
--webserver_port=0
--tserver_master_addrs=127.6.234.254:36373
--builtin_ntp_servers=127.6.234.212:44425
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
W20260620 11:46:20.828359 26925 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 11:46:20.828603 26925 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 11:46:20.828639 26925 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 11:46:20.829844 26925 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 11:46:20.829902 26925 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 11:46:20.829944 26925 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.6.234.194
I20260620 11:46:20.831292 26925 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.6.234.212:44425
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task1JRg7N/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781955746769051-7083-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-task1JRg7N/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781955746769051-7083-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.6.234.194:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task1JRg7N/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781955746769051-7083-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.6.234.194
--webserver_port=0
--tserver_master_addrs=127.6.234.254:36373
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.26925
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.6.234.194
--log_dir=/tmp/dist-test-task1JRg7N/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781955746769051-7083-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 20 Jun 2026 10:32:37 UTC on ccycloud.quasar-bgyofv.root.comops.site
build id 4457173
I20260620 11:46:20.831553 26925 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 11:46:20.831856 26925 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 11:46:20.836712 26933 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 11:46:20.836727 26931 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 11:46:20.836727 26930 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 11:46:20.837249 26925 server_base.cc:1034] running on GCE node
I20260620 11:46:20.837496 26925 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 11:46:20.837777 26925 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 11:46:20.838975 26925 hybrid_clock.cc:648] HybridClock initialized: now 1781955980838965 us; error 37 us; skew 500 ppm
I20260620 11:46:20.840176 26925 webserver.cc:460] Webserver started at http://127.6.234.194:37941/ using document root <none> and password file <none>
I20260620 11:46:20.840417 26925 fs_manager.cc:359] Metadata directory not provided
I20260620 11:46:20.840492 26925 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 11:46:20.840626 26925 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 11:46:20.841672 26925 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task1JRg7N/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781955746769051-7083-0/minicluster-data/ts-1/data/instance:
uuid: "782d45f75f91419fbdeb240f2332caef"
format_stamp: "Formatted at 2026-06-20 11:46:20 on dist-test-slave-cjc6"
server_key: "f604a92b29179aacdd51fcf3b8746cae"
server_key_iv: "46d4c08a83876b3253555156d55e0b5b"
server_key_version: "encryptionkey@0"
I20260620 11:46:20.842109 26925 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task1JRg7N/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781955746769051-7083-0/minicluster-data/ts-1/wal/instance:
uuid: "782d45f75f91419fbdeb240f2332caef"
format_stamp: "Formatted at 2026-06-20 11:46:20 on dist-test-slave-cjc6"
server_key: "f604a92b29179aacdd51fcf3b8746cae"
server_key_iv: "46d4c08a83876b3253555156d55e0b5b"
server_key_version: "encryptionkey@0"
I20260620 11:46:20.843354 26925 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.001s sys 0.000s
I20260620 11:46:20.844017 26939 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 11:46:20.844188 26925 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260620 11:46:20.844285 26925 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task1JRg7N/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781955746769051-7083-0/minicluster-data/ts-1/data,/tmp/dist-test-task1JRg7N/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781955746769051-7083-0/minicluster-data/ts-1/wal
uuid: "782d45f75f91419fbdeb240f2332caef"
format_stamp: "Formatted at 2026-06-20 11:46:20 on dist-test-slave-cjc6"
server_key: "f604a92b29179aacdd51fcf3b8746cae"
server_key_iv: "46d4c08a83876b3253555156d55e0b5b"
server_key_version: "encryptionkey@0"
I20260620 11:46:20.844364 26925 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task1JRg7N/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781955746769051-7083-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-task1JRg7N/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781955746769051-7083-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-task1JRg7N/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781955746769051-7083-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 11:46:20.867651 26925 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 11:46:20.868002 26925 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 11:46:20.868166 26925 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 11:46:20.868412 26925 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 11:46:20.868757 26925 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 11:46:20.868814 26925 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 11:46:20.868850 26925 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 11:46:20.868909 26925 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 11:46:20.874693 26925 rpc_server.cc:307] RPC server started. Bound to: 127.6.234.194:36433
I20260620 11:46:20.874740 27052 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.6.234.194:36433 every 8 connection(s)
I20260620 11:46:20.875068 26925 server_base.cc:1166] Dumped server information to /tmp/dist-test-task1JRg7N/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781955746769051-7083-0/minicluster-data/ts-1/data/info.pb
I20260620 11:46:20.879974 26773 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 11:46:20.875800 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.6.234.194:53671 (local address 127.6.234.254:36373)
0620 11:46:20.875860 (+ 60us) server_negotiation.cc:206] Beginning negotiation
0620 11:46:20.875863 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0620 11:46:20.876104 (+ 241us) server_negotiation.cc:407] Connection header received
0620 11:46:20.876561 (+ 457us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 11:46:20.876562 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 11:46:20.876571 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 11:46:20.876623 (+ 52us) server_negotiation.cc:226] Negotiated authn=SASL
0620 11:46:20.877265 (+ 642us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 11:46:20.877765 (+ 500us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 11:46:20.878695 (+ 930us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 11:46:20.878900 (+ 205us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 11:46:20.878958 (+ 58us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 11:46:20.879742 (+ 784us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 11:46:20.879743 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 11:46:20.879744 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 11:46:20.879747 (+ 3us) server_negotiation.cc:922] calling sasl_server_start()
0620 11:46:20.879761 (+ 14us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 11:46:20.879764 (+ 3us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 11:46:20.879805 (+ 41us) server_negotiation.cc:1035] Waiting for connection context
0620 11:46:20.879898 (+ 93us) server_negotiation.cc:299] Negotiation successful
0620 11:46:20.879918 (+ 20us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":23}
I20260620 11:46:20.880190 27055 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 11:46:20.875789 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.234.254:36373 (local address 127.6.234.194:53671)
0620 11:46:20.875976 (+ 187us) negotiation.cc:107] Waiting for socket to connect
0620 11:46:20.876012 (+ 36us) client_negotiation.cc:174] Beginning negotiation
0620 11:46:20.876479 (+ 467us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 11:46:20.876678 (+ 199us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 11:46:20.876681 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 11:46:20.876688 (+ 7us) client_negotiation.cc:189] Negotiated authn=SASL
0620 11:46:20.877193 (+ 505us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 11:46:20.877197 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 11:46:20.877831 (+ 634us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 11:46:20.877832 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 11:46:20.878612 (+ 780us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 11:46:20.878614 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 11:46:20.878994 (+ 380us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 11:46:20.878995 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 11:46:20.879112 (+ 117us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 11:46:20.879613 (+ 501us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0620 11:46:20.879614 (+ 1us) client_negotiation.cc:656] calling sasl_client_start()
0620 11:46:20.879641 (+ 27us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0620 11:46:20.879663 (+ 22us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 11:46:20.879817 (+ 154us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 11:46:20.879820 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 11:46:20.879822 (+ 2us) client_negotiation.cc:769] Sending connection context
0620 11:46:20.879872 (+ 50us) client_negotiation.cc:240] Negotiation successful
0620 11:46:20.879968 (+ 96us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":127,"thread_start_us":88,"threads_started":1}
I20260620 11:46:20.880695 27053 heartbeater.cc:347] Connected to a master server at 127.6.234.254:36373
I20260620 11:46:20.880810 27053 heartbeater.cc:464] Registering TS with master...
I20260620 11:46:20.881008 27053 heartbeater.cc:511] Master 127.6.234.254:36373 requested a full tablet report, sending...
I20260620 11:46:20.881378 26734 ts_manager.cc:194] Registered new tserver with Master: 782d45f75f91419fbdeb240f2332caef (127.6.234.194:36433)
I20260620 11:46:20.881769 26734 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.6.234.194:53671
I20260620 11:46:20.883302 7083 external_mini_cluster.cc:1388] Started /tmp/dist-test-task1JRg7N/build/release/bin/kudu as pid 26925
I20260620 11:46:20.883417 7083 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task1JRg7N/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781955746769051-7083-0/minicluster-data/ts-1/wal/instance
I20260620 11:46:20.883622 7083 external_mini_cluster.cc:1428] Setting key dc2e8301033db086f77bd6d9925e4684
I20260620 11:46:20.884514 7083 external_mini_cluster.cc:1326] Running /tmp/dist-test-task1JRg7N/build/release/bin/kudu
/tmp/dist-test-task1JRg7N/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task1JRg7N/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781955746769051-7083-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-task1JRg7N/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781955746769051-7083-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task1JRg7N/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781955746769051-7083-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-task1JRg7N/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781955746769051-7083-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.6.234.195:0
--local_ip_for_outbound_sockets=127.6.234.195
--webserver_interface=127.6.234.195
--webserver_port=0
--tserver_master_addrs=127.6.234.254:36373
--builtin_ntp_servers=127.6.234.212:44425
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
W20260620 11:46:20.965318 27056 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 11:46:20.965529 27056 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 11:46:20.965550 27056 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 11:46:20.966732 27056 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 11:46:20.966774 27056 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 11:46:20.966799 27056 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.6.234.195
I20260620 11:46:20.968122 27056 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.6.234.212:44425
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task1JRg7N/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781955746769051-7083-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-task1JRg7N/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781955746769051-7083-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.6.234.195:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task1JRg7N/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781955746769051-7083-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.6.234.195
--webserver_port=0
--tserver_master_addrs=127.6.234.254:36373
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.27056
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.6.234.195
--log_dir=/tmp/dist-test-task1JRg7N/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781955746769051-7083-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 20 Jun 2026 10:32:37 UTC on ccycloud.quasar-bgyofv.root.comops.site
build id 4457173
I20260620 11:46:20.968361 27056 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 11:46:20.968655 27056 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 11:46:20.973536 27062 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 11:46:20.973527 27064 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 11:46:20.973527 27061 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 11:46:20.973820 27056 server_base.cc:1034] running on GCE node
I20260620 11:46:20.974086 27056 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 11:46:20.974350 27056 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 11:46:20.975510 27056 hybrid_clock.cc:648] HybridClock initialized: now 1781955980975486 us; error 41 us; skew 500 ppm
I20260620 11:46:20.977061 27056 webserver.cc:460] Webserver started at http://127.6.234.195:42041/ using document root <none> and password file <none>
I20260620 11:46:20.977330 27056 fs_manager.cc:359] Metadata directory not provided
I20260620 11:46:20.977387 27056 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 11:46:20.977540 27056 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 11:46:20.978783 27056 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task1JRg7N/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781955746769051-7083-0/minicluster-data/ts-2/data/instance:
uuid: "3825a37f259642ed8372c636263543ba"
format_stamp: "Formatted at 2026-06-20 11:46:20 on dist-test-slave-cjc6"
server_key: "77afc804b6dd31fbd5f756e1e5487e44"
server_key_iv: "25962d72754413c5c8c69b2dfe9542cc"
server_key_version: "encryptionkey@0"
I20260620 11:46:20.979247 27056 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task1JRg7N/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781955746769051-7083-0/minicluster-data/ts-2/wal/instance:
uuid: "3825a37f259642ed8372c636263543ba"
format_stamp: "Formatted at 2026-06-20 11:46:20 on dist-test-slave-cjc6"
server_key: "77afc804b6dd31fbd5f756e1e5487e44"
server_key_iv: "25962d72754413c5c8c69b2dfe9542cc"
server_key_version: "encryptionkey@0"
I20260620 11:46:20.980674 27056 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260620 11:46:20.981529 27070 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 11:46:20.981751 27056 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260620 11:46:20.981846 27056 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task1JRg7N/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781955746769051-7083-0/minicluster-data/ts-2/data,/tmp/dist-test-task1JRg7N/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781955746769051-7083-0/minicluster-data/ts-2/wal
uuid: "3825a37f259642ed8372c636263543ba"
format_stamp: "Formatted at 2026-06-20 11:46:20 on dist-test-slave-cjc6"
server_key: "77afc804b6dd31fbd5f756e1e5487e44"
server_key_iv: "25962d72754413c5c8c69b2dfe9542cc"
server_key_version: "encryptionkey@0"
I20260620 11:46:20.981906 27056 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task1JRg7N/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781955746769051-7083-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-task1JRg7N/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781955746769051-7083-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-task1JRg7N/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781955746769051-7083-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 11:46:20.991317 27056 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 11:46:20.991684 27056 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 11:46:20.991819 27056 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 11:46:20.992009 27056 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 11:46:20.992308 27056 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 11:46:20.992342 27056 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 11:46:20.992365 27056 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 11:46:20.992379 27056 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 11:46:20.998421 27056 rpc_server.cc:307] RPC server started. Bound to: 127.6.234.195:44325
I20260620 11:46:20.998499 27183 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.6.234.195:44325 every 8 connection(s)
I20260620 11:46:20.998860 27056 server_base.cc:1166] Dumped server information to /tmp/dist-test-task1JRg7N/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781955746769051-7083-0/minicluster-data/ts-2/data/info.pb
I20260620 11:46:20.999365 7083 external_mini_cluster.cc:1388] Started /tmp/dist-test-task1JRg7N/build/release/bin/kudu as pid 27056
I20260620 11:46:20.999485 7083 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task1JRg7N/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781955746769051-7083-0/minicluster-data/ts-2/wal/instance
I20260620 11:46:20.999693 7083 external_mini_cluster.cc:1428] Setting key 5d85e22e9cf71bd1ffdd7ccbcf62546e
I20260620 11:46:21.004915 26773 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 11:46:21.000250 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.6.234.195:57631 (local address 127.6.234.254:36373)
0620 11:46:21.000306 (+ 56us) server_negotiation.cc:206] Beginning negotiation
0620 11:46:21.000308 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 11:46:21.000578 (+ 270us) server_negotiation.cc:407] Connection header received
0620 11:46:21.001193 (+ 615us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 11:46:21.001194 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 11:46:21.001206 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 11:46:21.001257 (+ 51us) server_negotiation.cc:226] Negotiated authn=SASL
0620 11:46:21.001988 (+ 731us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 11:46:21.002522 (+ 534us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 11:46:21.003515 (+ 993us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 11:46:21.003725 (+ 210us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 11:46:21.003812 (+ 87us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 11:46:21.004642 (+ 830us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 11:46:21.004643 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 11:46:21.004644 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 11:46:21.004648 (+ 4us) server_negotiation.cc:922] calling sasl_server_start()
0620 11:46:21.004663 (+ 15us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 11:46:21.004667 (+ 4us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 11:46:21.004711 (+ 44us) server_negotiation.cc:1035] Waiting for connection context
0620 11:46:21.004811 (+ 100us) server_negotiation.cc:299] Negotiation successful
0620 11:46:21.004838 (+ 27us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":31}
I20260620 11:46:21.005165 27186 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 11:46:21.000250 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.234.254:36373 (local address 127.6.234.195:57631)
0620 11:46:21.000464 (+ 214us) negotiation.cc:107] Waiting for socket to connect
0620 11:46:21.000509 (+ 45us) client_negotiation.cc:174] Beginning negotiation
0620 11:46:21.001086 (+ 577us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 11:46:21.001324 (+ 238us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 11:46:21.001328 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 11:46:21.001338 (+ 10us) client_negotiation.cc:189] Negotiated authn=SASL
0620 11:46:21.001898 (+ 560us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 11:46:21.001903 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 11:46:21.002620 (+ 717us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 11:46:21.002621 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 11:46:21.003418 (+ 797us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 11:46:21.003420 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 11:46:21.003821 (+ 401us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 11:46:21.003823 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 11:46:21.003962 (+ 139us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 11:46:21.004491 (+ 529us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0620 11:46:21.004492 (+ 1us) client_negotiation.cc:656] calling sasl_client_start()
0620 11:46:21.004519 (+ 27us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0620 11:46:21.004542 (+ 23us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 11:46:21.004722 (+ 180us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 11:46:21.004726 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 11:46:21.004729 (+ 3us) client_negotiation.cc:769] Sending connection context
0620 11:46:21.004815 (+ 86us) client_negotiation.cc:240] Negotiation successful
0620 11:46:21.004914 (+ 99us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":127,"thread_start_us":98,"threads_started":1}
I20260620 11:46:21.005594 27184 heartbeater.cc:347] Connected to a master server at 127.6.234.254:36373
I20260620 11:46:21.005681 27184 heartbeater.cc:464] Registering TS with master...
I20260620 11:46:21.005877 27184 heartbeater.cc:511] Master 127.6.234.254:36373 requested a full tablet report, sending...
I20260620 11:46:21.006217 26734 ts_manager.cc:194] Registered new tserver with Master: 3825a37f259642ed8372c636263543ba (127.6.234.195:44325)
I20260620 11:46:21.006582 26734 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.6.234.195:57631
I20260620 11:46:21.011674 7083 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260620 11:46:21.015672 26773 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 11:46:21.013039 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38834 (local address 127.6.234.254:36373)
0620 11:46:21.013112 (+ 73us) server_negotiation.cc:206] Beginning negotiation
0620 11:46:21.013114 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 11:46:21.013151 (+ 37us) server_negotiation.cc:407] Connection header received
0620 11:46:21.013214 (+ 63us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 11:46:21.013215 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 11:46:21.013224 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 11:46:21.013269 (+ 45us) server_negotiation.cc:226] Negotiated authn=SASL
0620 11:46:21.013401 (+ 132us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 11:46:21.013924 (+ 523us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 11:46:21.014631 (+ 707us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 11:46:21.014804 (+ 173us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 11:46:21.014856 (+ 52us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 11:46:21.015422 (+ 566us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 11:46:21.015424 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 11:46:21.015424 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 11:46:21.015453 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0620 11:46:21.015466 (+ 13us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 11:46:21.015470 (+ 4us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 11:46:21.015512 (+ 42us) server_negotiation.cc:1035] Waiting for connection context
0620 11:46:21.015560 (+ 48us) server_negotiation.cc:299] Negotiation successful
0620 11:46:21.015578 (+ 18us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":44}
I20260620 11:46:21.017113 26734 catalog_manager.cc:1908] Servicing CreateTable request from {username='slave'} at 127.0.0.1:38834:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260620 11:46:21.018685 26734 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260620 11:46:21.025310 27197 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 11:46:21.021169 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36098 (local address 127.6.234.195:44325)
0620 11:46:21.021332 (+ 163us) server_negotiation.cc:206] Beginning negotiation
0620 11:46:21.021336 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 11:46:21.021726 (+ 390us) server_negotiation.cc:407] Connection header received
0620 11:46:21.021922 (+ 196us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 11:46:21.021924 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 11:46:21.021940 (+ 16us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 11:46:21.022031 (+ 91us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 11:46:21.022209 (+ 178us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 11:46:21.023257 (+ 1048us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 11:46:21.024473 (+ 1216us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 11:46:21.024949 (+ 476us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 11:46:21.025052 (+ 103us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 11:46:21.025137 (+ 85us) server_negotiation.cc:1035] Waiting for connection context
0620 11:46:21.025191 (+ 54us) server_negotiation.cc:299] Negotiation successful
0620 11:46:21.025226 (+ 35us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":123,"thread_start_us":93,"threads_started":1}
I20260620 11:46:21.025313 27192 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 11:46:21.020779 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.234.193:34367 (local address 127.0.0.1:54344)
0620 11:46:21.020927 (+ 148us) negotiation.cc:107] Waiting for socket to connect
0620 11:46:21.020994 (+ 67us) client_negotiation.cc:174] Beginning negotiation
0620 11:46:21.021064 (+ 70us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 11:46:21.021519 (+ 455us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 11:46:21.021521 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 11:46:21.021523 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 11:46:21.021583 (+ 60us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 11:46:21.021584 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 11:46:21.022947 (+ 1363us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 11:46:21.022948 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 11:46:21.024071 (+ 1123us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 11:46:21.024073 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 11:46:21.025022 (+ 949us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 11:46:21.025023 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 11:46:21.025095 (+ 72us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 11:46:21.025107 (+ 12us) client_negotiation.cc:769] Sending connection context
0620 11:46:21.025155 (+ 48us) client_negotiation.cc:240] Negotiation successful
0620 11:46:21.025208 (+ 53us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":120,"thread_start_us":93,"threads_started":1}
I20260620 11:46:21.025309 27193 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 11:46:21.020820 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.234.195:44325 (local address 127.0.0.1:36098)
0620 11:46:21.021608 (+ 788us) negotiation.cc:107] Waiting for socket to connect
0620 11:46:21.021619 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0620 11:46:21.021675 (+ 56us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 11:46:21.022033 (+ 358us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 11:46:21.022034 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 11:46:21.022036 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 11:46:21.022077 (+ 41us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 11:46:21.022079 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 11:46:21.023362 (+ 1283us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 11:46:21.023363 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 11:46:21.024396 (+ 1033us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 11:46:21.024398 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 11:46:21.025057 (+ 659us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 11:46:21.025058 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 11:46:21.025118 (+ 60us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 11:46:21.025122 (+ 4us) client_negotiation.cc:769] Sending connection context
0620 11:46:21.025176 (+ 54us) client_negotiation.cc:240] Negotiation successful
0620 11:46:21.025218 (+ 42us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":751,"mutex_wait_us":16,"thread_start_us":98,"threads_started":1}
I20260620 11:46:21.025503 27196 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 11:46:21.021142 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35210 (local address 127.6.234.194:36433)
0620 11:46:21.021322 (+ 180us) server_negotiation.cc:206] Beginning negotiation
0620 11:46:21.021327 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 11:46:21.021870 (+ 543us) server_negotiation.cc:407] Connection header received
0620 11:46:21.021919 (+ 49us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 11:46:21.021922 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 11:46:21.021948 (+ 26us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 11:46:21.022025 (+ 77us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 11:46:21.022196 (+ 171us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 11:46:21.022959 (+ 763us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 11:46:21.024285 (+ 1326us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 11:46:21.024967 (+ 682us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 11:46:21.025030 (+ 63us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 11:46:21.025126 (+ 96us) server_negotiation.cc:1035] Waiting for connection context
0620 11:46:21.025350 (+ 224us) server_negotiation.cc:299] Negotiation successful
0620 11:46:21.025396 (+ 46us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":143,"thread_start_us":109,"threads_started":1}
I20260620 11:46:21.025575 27195 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 11:46:21.021021 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.234.194:36433 (local address 127.0.0.1:35210)
0620 11:46:21.021756 (+ 735us) negotiation.cc:107] Waiting for socket to connect
0620 11:46:21.021765 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0620 11:46:21.021812 (+ 47us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 11:46:21.022088 (+ 276us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 11:46:21.022088 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 11:46:21.022090 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 11:46:21.022119 (+ 29us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 11:46:21.022120 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 11:46:21.023046 (+ 926us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 11:46:21.023048 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 11:46:21.024198 (+ 1150us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 11:46:21.024200 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 11:46:21.025158 (+ 958us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 11:46:21.025159 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 11:46:21.025210 (+ 51us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 11:46:21.025214 (+ 4us) client_negotiation.cc:769] Sending connection context
0620 11:46:21.025262 (+ 48us) client_negotiation.cc:240] Negotiation successful
0620 11:46:21.025297 (+ 35us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":720,"mutex_wait_us":1,"thread_start_us":62,"threads_started":1}
I20260620 11:46:21.025799 27194 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 11:46:21.020942 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54344 (local address 127.6.234.193:34367)
0620 11:46:21.021372 (+ 430us) server_negotiation.cc:206] Beginning negotiation
0620 11:46:21.021375 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0620 11:46:21.021385 (+ 10us) server_negotiation.cc:407] Connection header received
0620 11:46:21.021429 (+ 44us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 11:46:21.021430 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 11:46:21.021442 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 11:46:21.021511 (+ 69us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 11:46:21.021688 (+ 177us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 11:46:21.022850 (+ 1162us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 11:46:21.024161 (+ 1311us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 11:46:21.024896 (+ 735us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 11:46:21.025512 (+ 616us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 11:46:21.025609 (+ 97us) server_negotiation.cc:1035] Waiting for connection context
0620 11:46:21.025637 (+ 28us) server_negotiation.cc:299] Negotiation successful
0620 11:46:21.025685 (+ 48us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":396,"thread_start_us":118,"threads_started":1}
I20260620 11:46:21.026432 26856 tablet_service.cc:1467] Processing CreateTablet for tablet d2f02df05b874b5a9d9a7c264b0b4d2e (DEFAULT_TABLE table=test-table [id=4cfab08479864c708fe4a0a5a6b5a089]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 11:46:21.026432 26987 tablet_service.cc:1467] Processing CreateTablet for tablet d2f02df05b874b5a9d9a7c264b0b4d2e (DEFAULT_TABLE table=test-table [id=4cfab08479864c708fe4a0a5a6b5a089]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 11:46:21.026794 26856 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d2f02df05b874b5a9d9a7c264b0b4d2e. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 11:46:21.026794 26987 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d2f02df05b874b5a9d9a7c264b0b4d2e. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 11:46:21.030180 27198 tablet_bootstrap.cc:492] T d2f02df05b874b5a9d9a7c264b0b4d2e P 782d45f75f91419fbdeb240f2332caef: Bootstrap starting.
I20260620 11:46:21.030791 27198 tablet_bootstrap.cc:654] T d2f02df05b874b5a9d9a7c264b0b4d2e P 782d45f75f91419fbdeb240f2332caef: Neither blocks nor log segments found. Creating new log.
I20260620 11:46:21.030786 27118 tablet_service.cc:1467] Processing CreateTablet for tablet d2f02df05b874b5a9d9a7c264b0b4d2e (DEFAULT_TABLE table=test-table [id=4cfab08479864c708fe4a0a5a6b5a089]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 11:46:21.031136 27118 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d2f02df05b874b5a9d9a7c264b0b4d2e. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 11:46:21.031136 27198 log.cc:824] T d2f02df05b874b5a9d9a7c264b0b4d2e P 782d45f75f91419fbdeb240f2332caef: Log is configured to *not* fsync() on all Append() calls
I20260620 11:46:21.031509 27199 tablet_bootstrap.cc:492] T d2f02df05b874b5a9d9a7c264b0b4d2e P f6d343cd73fd43b7a7c7f2bc124b0fa9: Bootstrap starting.
I20260620 11:46:21.031951 27198 tablet_bootstrap.cc:492] T d2f02df05b874b5a9d9a7c264b0b4d2e P 782d45f75f91419fbdeb240f2332caef: No bootstrap required, opened a new log
I20260620 11:46:21.032059 27198 ts_tablet_manager.cc:1397] T d2f02df05b874b5a9d9a7c264b0b4d2e P 782d45f75f91419fbdeb240f2332caef: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20260620 11:46:21.032083 27199 tablet_bootstrap.cc:654] T d2f02df05b874b5a9d9a7c264b0b4d2e P f6d343cd73fd43b7a7c7f2bc124b0fa9: Neither blocks nor log segments found. Creating new log.
I20260620 11:46:21.032469 27199 log.cc:824] T d2f02df05b874b5a9d9a7c264b0b4d2e P f6d343cd73fd43b7a7c7f2bc124b0fa9: Log is configured to *not* fsync() on all Append() calls
I20260620 11:46:21.033140 27199 tablet_bootstrap.cc:492] T d2f02df05b874b5a9d9a7c264b0b4d2e P f6d343cd73fd43b7a7c7f2bc124b0fa9: No bootstrap required, opened a new log
I20260620 11:46:21.033223 27199 ts_tablet_manager.cc:1397] T d2f02df05b874b5a9d9a7c264b0b4d2e P f6d343cd73fd43b7a7c7f2bc124b0fa9: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260620 11:46:21.034487 27202 tablet_bootstrap.cc:492] T d2f02df05b874b5a9d9a7c264b0b4d2e P 3825a37f259642ed8372c636263543ba: Bootstrap starting.
I20260620 11:46:21.034870 27198 raft_consensus.cc:348] T d2f02df05b874b5a9d9a7c264b0b4d2e P 782d45f75f91419fbdeb240f2332caef [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f6d343cd73fd43b7a7c7f2bc124b0fa9" member_type: VOTER last_known_addr { host: "127.6.234.193" port: 34367 } } peers { permanent_uuid: "3825a37f259642ed8372c636263543ba" member_type: VOTER last_known_addr { host: "127.6.234.195" port: 44325 } } peers { permanent_uuid: "782d45f75f91419fbdeb240f2332caef" member_type: VOTER last_known_addr { host: "127.6.234.194" port: 36433 } }
I20260620 11:46:21.035000 27198 raft_consensus.cc:374] T d2f02df05b874b5a9d9a7c264b0b4d2e P 782d45f75f91419fbdeb240f2332caef [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 11:46:21.035053 27198 raft_consensus.cc:729] T d2f02df05b874b5a9d9a7c264b0b4d2e P 782d45f75f91419fbdeb240f2332caef [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 782d45f75f91419fbdeb240f2332caef, State: Initialized, Role: FOLLOWER
I20260620 11:46:21.035187 27198 consensus_queue.cc:260] T d2f02df05b874b5a9d9a7c264b0b4d2e P 782d45f75f91419fbdeb240f2332caef [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f6d343cd73fd43b7a7c7f2bc124b0fa9" member_type: VOTER last_known_addr { host: "127.6.234.193" port: 34367 } } peers { permanent_uuid: "3825a37f259642ed8372c636263543ba" member_type: VOTER last_known_addr { host: "127.6.234.195" port: 44325 } } peers { permanent_uuid: "782d45f75f91419fbdeb240f2332caef" member_type: VOTER last_known_addr { host: "127.6.234.194" port: 36433 } }
I20260620 11:46:21.035230 27202 tablet_bootstrap.cc:654] T d2f02df05b874b5a9d9a7c264b0b4d2e P 3825a37f259642ed8372c636263543ba: Neither blocks nor log segments found. Creating new log.
I20260620 11:46:21.035444 27198 ts_tablet_manager.cc:1428] T d2f02df05b874b5a9d9a7c264b0b4d2e P 782d45f75f91419fbdeb240f2332caef: Time spent starting tablet: real 0.003s user 0.003s sys 0.000s
I20260620 11:46:21.035578 27053 heartbeater.cc:503] Master 127.6.234.254:36373 was elected leader, sending a full tablet report...
I20260620 11:46:21.035629 27202 log.cc:824] T d2f02df05b874b5a9d9a7c264b0b4d2e P 3825a37f259642ed8372c636263543ba: Log is configured to *not* fsync() on all Append() calls
I20260620 11:46:21.035591 27199 raft_consensus.cc:348] T d2f02df05b874b5a9d9a7c264b0b4d2e P f6d343cd73fd43b7a7c7f2bc124b0fa9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f6d343cd73fd43b7a7c7f2bc124b0fa9" member_type: VOTER last_known_addr { host: "127.6.234.193" port: 34367 } } peers { permanent_uuid: "3825a37f259642ed8372c636263543ba" member_type: VOTER last_known_addr { host: "127.6.234.195" port: 44325 } } peers { permanent_uuid: "782d45f75f91419fbdeb240f2332caef" member_type: VOTER last_known_addr { host: "127.6.234.194" port: 36433 } }
I20260620 11:46:21.035742 27199 raft_consensus.cc:374] T d2f02df05b874b5a9d9a7c264b0b4d2e P f6d343cd73fd43b7a7c7f2bc124b0fa9 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 11:46:21.035794 27199 raft_consensus.cc:729] T d2f02df05b874b5a9d9a7c264b0b4d2e P f6d343cd73fd43b7a7c7f2bc124b0fa9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f6d343cd73fd43b7a7c7f2bc124b0fa9, State: Initialized, Role: FOLLOWER
I20260620 11:46:21.035931 27199 consensus_queue.cc:260] T d2f02df05b874b5a9d9a7c264b0b4d2e P f6d343cd73fd43b7a7c7f2bc124b0fa9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f6d343cd73fd43b7a7c7f2bc124b0fa9" member_type: VOTER last_known_addr { host: "127.6.234.193" port: 34367 } } peers { permanent_uuid: "3825a37f259642ed8372c636263543ba" member_type: VOTER last_known_addr { host: "127.6.234.195" port: 44325 } } peers { permanent_uuid: "782d45f75f91419fbdeb240f2332caef" member_type: VOTER last_known_addr { host: "127.6.234.194" port: 36433 } }
I20260620 11:46:21.036196 27199 ts_tablet_manager.cc:1428] T d2f02df05b874b5a9d9a7c264b0b4d2e P f6d343cd73fd43b7a7c7f2bc124b0fa9: Time spent starting tablet: real 0.003s user 0.003s sys 0.000s
I20260620 11:46:21.036269 26922 heartbeater.cc:503] Master 127.6.234.254:36373 was elected leader, sending a full tablet report...
I20260620 11:46:21.037142 27202 tablet_bootstrap.cc:492] T d2f02df05b874b5a9d9a7c264b0b4d2e P 3825a37f259642ed8372c636263543ba: No bootstrap required, opened a new log
I20260620 11:46:21.037235 27202 ts_tablet_manager.cc:1397] T d2f02df05b874b5a9d9a7c264b0b4d2e P 3825a37f259642ed8372c636263543ba: Time spent bootstrapping tablet: real 0.003s user 0.001s sys 0.000s
I20260620 11:46:21.039047 27202 raft_consensus.cc:348] T d2f02df05b874b5a9d9a7c264b0b4d2e P 3825a37f259642ed8372c636263543ba [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f6d343cd73fd43b7a7c7f2bc124b0fa9" member_type: VOTER last_known_addr { host: "127.6.234.193" port: 34367 } } peers { permanent_uuid: "3825a37f259642ed8372c636263543ba" member_type: VOTER last_known_addr { host: "127.6.234.195" port: 44325 } } peers { permanent_uuid: "782d45f75f91419fbdeb240f2332caef" member_type: VOTER last_known_addr { host: "127.6.234.194" port: 36433 } }
I20260620 11:46:21.039166 27202 raft_consensus.cc:374] T d2f02df05b874b5a9d9a7c264b0b4d2e P 3825a37f259642ed8372c636263543ba [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 11:46:21.039188 27202 raft_consensus.cc:729] T d2f02df05b874b5a9d9a7c264b0b4d2e P 3825a37f259642ed8372c636263543ba [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3825a37f259642ed8372c636263543ba, State: Initialized, Role: FOLLOWER
I20260620 11:46:21.039291 27202 consensus_queue.cc:260] T d2f02df05b874b5a9d9a7c264b0b4d2e P 3825a37f259642ed8372c636263543ba [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f6d343cd73fd43b7a7c7f2bc124b0fa9" member_type: VOTER last_known_addr { host: "127.6.234.193" port: 34367 } } peers { permanent_uuid: "3825a37f259642ed8372c636263543ba" member_type: VOTER last_known_addr { host: "127.6.234.195" port: 44325 } } peers { permanent_uuid: "782d45f75f91419fbdeb240f2332caef" member_type: VOTER last_known_addr { host: "127.6.234.194" port: 36433 } }
I20260620 11:46:21.039490 27202 ts_tablet_manager.cc:1428] T d2f02df05b874b5a9d9a7c264b0b4d2e P 3825a37f259642ed8372c636263543ba: Time spent starting tablet: real 0.002s user 0.003s sys 0.000s
I20260620 11:46:21.039599 27184 heartbeater.cc:503] Master 127.6.234.254:36373 was elected leader, sending a full tablet report...
I20260620 11:46:21.060752 27205 raft_consensus.cc:482] T d2f02df05b874b5a9d9a7c264b0b4d2e P f6d343cd73fd43b7a7c7f2bc124b0fa9 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260620 11:46:21.060890 27205 raft_consensus.cc:504] T d2f02df05b874b5a9d9a7c264b0b4d2e P f6d343cd73fd43b7a7c7f2bc124b0fa9 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f6d343cd73fd43b7a7c7f2bc124b0fa9" member_type: VOTER last_known_addr { host: "127.6.234.193" port: 34367 } } peers { permanent_uuid: "3825a37f259642ed8372c636263543ba" member_type: VOTER last_known_addr { host: "127.6.234.195" port: 44325 } } peers { permanent_uuid: "782d45f75f91419fbdeb240f2332caef" member_type: VOTER last_known_addr { host: "127.6.234.194" port: 36433 } }
I20260620 11:46:21.061240 27205 leader_election.cc:288] T d2f02df05b874b5a9d9a7c264b0b4d2e P f6d343cd73fd43b7a7c7f2bc124b0fa9 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 3825a37f259642ed8372c636263543ba (127.6.234.195:44325), 782d45f75f91419fbdeb240f2332caef (127.6.234.194:36433)
I20260620 11:46:21.065258 27196 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 11:46:21.061524 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.6.234.193:40905 (local address 127.6.234.194:36433)
0620 11:46:21.061574 (+ 50us) server_negotiation.cc:206] Beginning negotiation
0620 11:46:21.061577 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0620 11:46:21.061724 (+ 147us) server_negotiation.cc:407] Connection header received
0620 11:46:21.061780 (+ 56us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 11:46:21.061781 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 11:46:21.061789 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 11:46:21.061837 (+ 48us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 11:46:21.061990 (+ 153us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 11:46:21.062700 (+ 710us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 11:46:21.064313 (+ 1613us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 11:46:21.064882 (+ 569us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 11:46:21.064970 (+ 88us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 11:46:21.064977 (+ 7us) server_negotiation.cc:1035] Waiting for connection context
0620 11:46:21.065139 (+ 162us) server_negotiation.cc:299] Negotiation successful
0620 11:46:21.065183 (+ 44us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":25}
I20260620 11:46:21.065258 27197 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 11:46:21.061516 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.6.234.193:58845 (local address 127.6.234.195:44325)
0620 11:46:21.061578 (+ 62us) server_negotiation.cc:206] Beginning negotiation
0620 11:46:21.061580 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 11:46:21.061588 (+ 8us) server_negotiation.cc:407] Connection header received
0620 11:46:21.061625 (+ 37us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 11:46:21.061625 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 11:46:21.061633 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 11:46:21.061734 (+ 101us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 11:46:21.061916 (+ 182us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 11:46:21.062700 (+ 784us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 11:46:21.064310 (+ 1610us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 11:46:21.064883 (+ 573us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 11:46:21.064965 (+ 82us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 11:46:21.064974 (+ 9us) server_negotiation.cc:1035] Waiting for connection context
0620 11:46:21.065143 (+ 169us) server_negotiation.cc:299] Negotiation successful
0620 11:46:21.065183 (+ 40us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":41}
I20260620 11:46:21.065258 26924 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 11:46:21.061376 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.234.195:44325 (local address 127.6.234.193:58845)
0620 11:46:21.061451 (+ 75us) negotiation.cc:107] Waiting for socket to connect
0620 11:46:21.061465 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0620 11:46:21.061513 (+ 48us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 11:46:21.061741 (+ 228us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 11:46:21.061742 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 11:46:21.061743 (+ 1us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 11:46:21.061838 (+ 95us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 11:46:21.061841 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 11:46:21.062791 (+ 950us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 11:46:21.062792 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 11:46:21.064206 (+ 1414us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 11:46:21.064210 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 11:46:21.064973 (+ 763us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 11:46:21.064974 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 11:46:21.065062 (+ 88us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 11:46:21.065064 (+ 2us) client_negotiation.cc:769] Sending connection context
0620 11:46:21.065120 (+ 56us) client_negotiation.cc:240] Negotiation successful
0620 11:46:21.065163 (+ 43us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":40,"mutex_wait_us":106}
I20260620 11:46:21.065264 27207 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 11:46:21.061469 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.234.194:36433 (local address 127.6.234.193:40905)
0620 11:46:21.061617 (+ 148us) negotiation.cc:107] Waiting for socket to connect
0620 11:46:21.061625 (+ 8us) client_negotiation.cc:174] Beginning negotiation
0620 11:46:21.061674 (+ 49us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 11:46:21.061858 (+ 184us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 11:46:21.061859 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 11:46:21.061861 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 11:46:21.061921 (+ 60us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 11:46:21.061922 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 11:46:21.062791 (+ 869us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 11:46:21.062792 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 11:46:21.064206 (+ 1414us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 11:46:21.064210 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 11:46:21.064974 (+ 764us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 11:46:21.064974 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 11:46:21.065053 (+ 79us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 11:46:21.065058 (+ 5us) client_negotiation.cc:769] Sending connection context
0620 11:46:21.065120 (+ 62us) client_negotiation.cc:240] Negotiation successful
0620 11:46:21.065162 (+ 42us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":132,"mutex_wait_us":81,"thread_start_us":108,"threads_started":1}
I20260620 11:46:21.065634 27138 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d2f02df05b874b5a9d9a7c264b0b4d2e" candidate_uuid: "f6d343cd73fd43b7a7c7f2bc124b0fa9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3825a37f259642ed8372c636263543ba" is_pre_election: true
I20260620 11:46:21.065842 27138 raft_consensus.cc:2454] T d2f02df05b874b5a9d9a7c264b0b4d2e P 3825a37f259642ed8372c636263543ba [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f6d343cd73fd43b7a7c7f2bc124b0fa9 in term 0.
I20260620 11:46:21.065924 27007 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d2f02df05b874b5a9d9a7c264b0b4d2e" candidate_uuid: "f6d343cd73fd43b7a7c7f2bc124b0fa9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "782d45f75f91419fbdeb240f2332caef" is_pre_election: true
I20260620 11:46:21.066079 27007 raft_consensus.cc:2454] T d2f02df05b874b5a9d9a7c264b0b4d2e P 782d45f75f91419fbdeb240f2332caef [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f6d343cd73fd43b7a7c7f2bc124b0fa9 in term 0.
I20260620 11:46:21.066107 26811 leader_election.cc:302] T d2f02df05b874b5a9d9a7c264b0b4d2e P f6d343cd73fd43b7a7c7f2bc124b0fa9 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3825a37f259642ed8372c636263543ba, f6d343cd73fd43b7a7c7f2bc124b0fa9; no voters:
I20260620 11:46:21.066248 27205 raft_consensus.cc:2781] T d2f02df05b874b5a9d9a7c264b0b4d2e P f6d343cd73fd43b7a7c7f2bc124b0fa9 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260620 11:46:21.066306 27205 raft_consensus.cc:482] T d2f02df05b874b5a9d9a7c264b0b4d2e P f6d343cd73fd43b7a7c7f2bc124b0fa9 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260620 11:46:21.066329 27205 raft_consensus.cc:3037] T d2f02df05b874b5a9d9a7c264b0b4d2e P f6d343cd73fd43b7a7c7f2bc124b0fa9 [term 0 FOLLOWER]: Advancing to term 1
I20260620 11:46:21.067031 27205 raft_consensus.cc:504] T d2f02df05b874b5a9d9a7c264b0b4d2e P f6d343cd73fd43b7a7c7f2bc124b0fa9 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f6d343cd73fd43b7a7c7f2bc124b0fa9" member_type: VOTER last_known_addr { host: "127.6.234.193" port: 34367 } } peers { permanent_uuid: "3825a37f259642ed8372c636263543ba" member_type: VOTER last_known_addr { host: "127.6.234.195" port: 44325 } } peers { permanent_uuid: "782d45f75f91419fbdeb240f2332caef" member_type: VOTER last_known_addr { host: "127.6.234.194" port: 36433 } }
I20260620 11:46:21.067183 27205 leader_election.cc:288] T d2f02df05b874b5a9d9a7c264b0b4d2e P f6d343cd73fd43b7a7c7f2bc124b0fa9 [CANDIDATE]: Term 1 election: Requested vote from peers 3825a37f259642ed8372c636263543ba (127.6.234.195:44325), 782d45f75f91419fbdeb240f2332caef (127.6.234.194:36433)
I20260620 11:46:21.067346 27138 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d2f02df05b874b5a9d9a7c264b0b4d2e" candidate_uuid: "f6d343cd73fd43b7a7c7f2bc124b0fa9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3825a37f259642ed8372c636263543ba"
I20260620 11:46:21.067378 27007 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d2f02df05b874b5a9d9a7c264b0b4d2e" candidate_uuid: "f6d343cd73fd43b7a7c7f2bc124b0fa9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "782d45f75f91419fbdeb240f2332caef"
I20260620 11:46:21.067444 27138 raft_consensus.cc:3037] T d2f02df05b874b5a9d9a7c264b0b4d2e P 3825a37f259642ed8372c636263543ba [term 0 FOLLOWER]: Advancing to term 1
I20260620 11:46:21.067451 27007 raft_consensus.cc:3037] T d2f02df05b874b5a9d9a7c264b0b4d2e P 782d45f75f91419fbdeb240f2332caef [term 0 FOLLOWER]: Advancing to term 1
I20260620 11:46:21.068164 27138 raft_consensus.cc:2454] T d2f02df05b874b5a9d9a7c264b0b4d2e P 3825a37f259642ed8372c636263543ba [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f6d343cd73fd43b7a7c7f2bc124b0fa9 in term 1.
I20260620 11:46:21.068233 27007 raft_consensus.cc:2454] T d2f02df05b874b5a9d9a7c264b0b4d2e P 782d45f75f91419fbdeb240f2332caef [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f6d343cd73fd43b7a7c7f2bc124b0fa9 in term 1.
I20260620 11:46:21.068367 26811 leader_election.cc:302] T d2f02df05b874b5a9d9a7c264b0b4d2e P f6d343cd73fd43b7a7c7f2bc124b0fa9 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3825a37f259642ed8372c636263543ba, f6d343cd73fd43b7a7c7f2bc124b0fa9; no voters:
I20260620 11:46:21.068495 27205 raft_consensus.cc:2781] T d2f02df05b874b5a9d9a7c264b0b4d2e P f6d343cd73fd43b7a7c7f2bc124b0fa9 [term 1 FOLLOWER]: Leader election won for term 1
I20260620 11:46:21.068692 27205 raft_consensus.cc:686] T d2f02df05b874b5a9d9a7c264b0b4d2e P f6d343cd73fd43b7a7c7f2bc124b0fa9 [term 1 LEADER]: Becoming Leader. State: Replica: f6d343cd73fd43b7a7c7f2bc124b0fa9, State: Running, Role: LEADER
I20260620 11:46:21.068907 27205 consensus_queue.cc:237] T d2f02df05b874b5a9d9a7c264b0b4d2e P f6d343cd73fd43b7a7c7f2bc124b0fa9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f6d343cd73fd43b7a7c7f2bc124b0fa9" member_type: VOTER last_known_addr { host: "127.6.234.193" port: 34367 } } peers { permanent_uuid: "3825a37f259642ed8372c636263543ba" member_type: VOTER last_known_addr { host: "127.6.234.195" port: 44325 } } peers { permanent_uuid: "782d45f75f91419fbdeb240f2332caef" member_type: VOTER last_known_addr { host: "127.6.234.194" port: 36433 } }
I20260620 11:46:21.069839 26732 catalog_manager.cc:5495] T d2f02df05b874b5a9d9a7c264b0b4d2e P f6d343cd73fd43b7a7c7f2bc124b0fa9 reported cstate change: term changed from 0 to 1, leader changed from <none> to f6d343cd73fd43b7a7c7f2bc124b0fa9 (127.6.234.193). New cstate: current_term: 1 leader_uuid: "f6d343cd73fd43b7a7c7f2bc124b0fa9" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f6d343cd73fd43b7a7c7f2bc124b0fa9" member_type: VOTER last_known_addr { host: "127.6.234.193" port: 34367 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "3825a37f259642ed8372c636263543ba" member_type: VOTER last_known_addr { host: "127.6.234.195" port: 44325 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "782d45f75f91419fbdeb240f2332caef" member_type: VOTER last_known_addr { host: "127.6.234.194" port: 36433 } health_report { overall_health: UNKNOWN } } }
I20260620 11:46:21.082597 27194 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 11:46:21.080197 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54348 (local address 127.6.234.193:34367)
0620 11:46:21.080256 (+ 59us) server_negotiation.cc:206] Beginning negotiation
0620 11:46:21.080257 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0620 11:46:21.080264 (+ 7us) server_negotiation.cc:407] Connection header received
0620 11:46:21.080293 (+ 29us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 11:46:21.080293 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 11:46:21.080310 (+ 17us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 11:46:21.080382 (+ 72us) server_negotiation.cc:226] Negotiated authn=TOKEN
0620 11:46:21.080529 (+ 147us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 11:46:21.081014 (+ 485us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 11:46:21.081797 (+ 783us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 11:46:21.081991 (+ 194us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 11:46:21.082074 (+ 83us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 11:46:21.082253 (+ 179us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0620 11:46:21.082354 (+ 101us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0620 11:46:21.082446 (+ 92us) server_negotiation.cc:1035] Waiting for connection context
0620 11:46:21.082471 (+ 25us) server_negotiation.cc:299] Negotiation successful
0620 11:46:21.082491 (+ 20us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":39}
I20260620 11:46:21.084432 27007 raft_consensus.cc:1261] T d2f02df05b874b5a9d9a7c264b0b4d2e P 782d45f75f91419fbdeb240f2332caef [term 1 FOLLOWER]: Refusing update from remote peer f6d343cd73fd43b7a7c7f2bc124b0fa9: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 11:46:21.084432 27138 raft_consensus.cc:1261] T d2f02df05b874b5a9d9a7c264b0b4d2e P 3825a37f259642ed8372c636263543ba [term 1 FOLLOWER]: Refusing update from remote peer f6d343cd73fd43b7a7c7f2bc124b0fa9: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 11:46:21.084813 27205 consensus_queue.cc:1035] T d2f02df05b874b5a9d9a7c264b0b4d2e P f6d343cd73fd43b7a7c7f2bc124b0fa9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "3825a37f259642ed8372c636263543ba" member_type: VOTER last_known_addr { host: "127.6.234.195" port: 44325 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 11:46:21.084981 27208 consensus_queue.cc:1035] T d2f02df05b874b5a9d9a7c264b0b4d2e P f6d343cd73fd43b7a7c7f2bc124b0fa9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "782d45f75f91419fbdeb240f2332caef" member_type: VOTER last_known_addr { host: "127.6.234.194" port: 36433 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 11:46:21.088215 27213 mvcc.cc:205] Tried to move back new op lower bound from 7298891698518970368 to 7298891698458980352. Current Snapshot: MvccSnapshot[applied={T|T < 7298891698518970368 or (T in {7298891698518970368})}]
I20260620 11:46:21.089922 26732 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:38834:
table { table_name: "test-table" } modify_external_catalogs: true
I20260620 11:46:21.090036 26732 catalog_manager.cc:2645] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:38834:
table { table_name: "test-table" } modify_external_catalogs: true
I20260620 11:46:21.090797 26732 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 17d389e1a8b0416db7b1743fc8696131: Sending DeleteTablet for 3 replicas of tablet d2f02df05b874b5a9d9a7c264b0b4d2e
I20260620 11:46:21.091121 26856 tablet_service.cc:1514] Processing DeleteTablet for tablet d2f02df05b874b5a9d9a7c264b0b4d2e with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 11:46:21 UTC) from {username='slave'} at 127.0.0.1:54344
I20260620 11:46:21.091197 26987 tablet_service.cc:1514] Processing DeleteTablet for tablet d2f02df05b874b5a9d9a7c264b0b4d2e with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 11:46:21 UTC) from {username='slave'} at 127.0.0.1:35210
I20260620 11:46:21.091374 27221 tablet_replica.cc:330] T d2f02df05b874b5a9d9a7c264b0b4d2e P 782d45f75f91419fbdeb240f2332caef: stopping tablet replica
I20260620 11:46:21.091467 27221 raft_consensus.cc:2229] T d2f02df05b874b5a9d9a7c264b0b4d2e P 782d45f75f91419fbdeb240f2332caef [term 1 FOLLOWER]: Raft consensus shutting down.
I20260620 11:46:21.091496 27118 tablet_service.cc:1514] Processing DeleteTablet for tablet d2f02df05b874b5a9d9a7c264b0b4d2e with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 11:46:21 UTC) from {username='slave'} at 127.0.0.1:36098
I20260620 11:46:21.091562 27221 raft_consensus.cc:2258] T d2f02df05b874b5a9d9a7c264b0b4d2e P 782d45f75f91419fbdeb240f2332caef [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 11:46:21.091735 7083 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task1JRg7N/build/release/bin/kudu with pid 26794
I20260620 11:46:21.093536 27222 tablet_replica.cc:330] T d2f02df05b874b5a9d9a7c264b0b4d2e P 3825a37f259642ed8372c636263543ba: stopping tablet replica
I20260620 11:46:21.093655 27222 raft_consensus.cc:2229] T d2f02df05b874b5a9d9a7c264b0b4d2e P 3825a37f259642ed8372c636263543ba [term 1 FOLLOWER]: Raft consensus shutting down.
I20260620 11:46:21.093782 27222 raft_consensus.cc:2258] T d2f02df05b874b5a9d9a7c264b0b4d2e P 3825a37f259642ed8372c636263543ba [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 11:46:21.094152 27222 ts_tablet_manager.cc:1905] T d2f02df05b874b5a9d9a7c264b0b4d2e P 3825a37f259642ed8372c636263543ba: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 11:46:21.095690 27222 ts_tablet_manager.cc:1918] T d2f02df05b874b5a9d9a7c264b0b4d2e P 3825a37f259642ed8372c636263543ba: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260620 11:46:21.095769 27222 log.cc:1196] T d2f02df05b874b5a9d9a7c264b0b4d2e P 3825a37f259642ed8372c636263543ba: Deleting WAL directory at /tmp/dist-test-task1JRg7N/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781955746769051-7083-0/minicluster-data/ts-2/wal/wals/d2f02df05b874b5a9d9a7c264b0b4d2e
I20260620 11:46:21.096060 27222 ts_tablet_manager.cc:1939] T d2f02df05b874b5a9d9a7c264b0b4d2e P 3825a37f259642ed8372c636263543ba: Deleting consensus metadata
I20260620 11:46:21.096148 27221 ts_tablet_manager.cc:1905] T d2f02df05b874b5a9d9a7c264b0b4d2e P 782d45f75f91419fbdeb240f2332caef: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 11:46:21.096463 26720 catalog_manager.cc:4841] TS 3825a37f259642ed8372c636263543ba (127.6.234.195:44325): tablet d2f02df05b874b5a9d9a7c264b0b4d2e (table test-table [id=4cfab08479864c708fe4a0a5a6b5a089]) successfully deleted
W20260620 11:46:21.097249 26718 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.6.234.193:34367 (error 108)
I20260620 11:46:21.097486 7083 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task1JRg7N/build/release/bin/kudu with pid 26925
I20260620 11:46:21.097689 27221 ts_tablet_manager.cc:1918] T d2f02df05b874b5a9d9a7c264b0b4d2e P 782d45f75f91419fbdeb240f2332caef: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260620 11:46:21.097788 27221 log.cc:1196] T d2f02df05b874b5a9d9a7c264b0b4d2e P 782d45f75f91419fbdeb240f2332caef: Deleting WAL directory at /tmp/dist-test-task1JRg7N/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781955746769051-7083-0/minicluster-data/ts-1/wal/wals/d2f02df05b874b5a9d9a7c264b0b4d2e
I20260620 11:46:21.098083 27195 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 11:46:21.097556 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.234.193:34367 (local address 127.0.0.1:54360)
0620 11:46:21.098006 (+ 450us) negotiation.cc:107] Waiting for socket to connect
0620 11:46:21.098056 (+ 50us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.234.193:34367: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":428}
W20260620 11:46:21.098202 26718 catalog_manager.cc:4584] TS f6d343cd73fd43b7a7c7f2bc124b0fa9 (127.6.234.193:34367): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet d2f02df05b874b5a9d9a7c264b0b4d2e: Network error: Client connection negotiation failed: client connection to 127.6.234.193:34367: connect: Connection refused (error 111)
I20260620 11:46:21.102475 7083 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task1JRg7N/build/release/bin/kudu with pid 27056
I20260620 11:46:21.105406 27195 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 11:46:21.104288 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.234.194:36433 (local address 127.0.0.1:35224)
0620 11:46:21.105349 (+ 1061us) negotiation.cc:107] Waiting for socket to connect
0620 11:46:21.105379 (+ 30us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.234.194:36433: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":1039}
I20260620 11:46:21.108635 7083 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task1JRg7N/build/release/bin/kudu with pid 26703
2026-06-20T11:46:21Z chronyd exiting
[ OK ] EncryptionPolicyTest.TestEncryptionPolicy/5 (674 ms)
[----------] 6 tests from EncryptionPolicyTest (8119 ms total)
[----------] 18 tests from AuthTokenIssuingTest
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/0
Loading random data
Initializing database '/tmp/dist-test-task1JRg7N/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1781955746769051-7083-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 20 11:46:21 dist-test-slave-cjc6 krb5kdc[27226](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 20 11:46:21 dist-test-slave-cjc6 krb5kdc[27226](info): set up 2 sockets
Jun 20 11:46:21 dist-test-slave-cjc6 krb5kdc[27226](info): commencing operation
krb5kdc: starting...
W20260620 11:46:22.145520 7083 mini_kdc.cc:121] Time spent starting KDC: real 1.017s user 0.000s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task1JRg7N/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1781955746769051-7083-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task1JRg7N/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1781955746769051-7083-0/krb5kdc/test-user.keytab.
Jun 20 11:46:22 dist-test-slave-cjc6 krb5kdc[27226](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781955982, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-20T11:46:22Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-20T11:46:22Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task1JRg7N/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1781955746769051-7083-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task1JRg7N/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1781955746769051-7083-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260620 11:46:22.260747 7083 external_mini_cluster.cc:1326] Running /tmp/dist-test-task1JRg7N/build/release/bin/kudu
/tmp/dist-test-task1JRg7N/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task1JRg7N/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1781955746769051-7083-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task1JRg7N/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1781955746769051-7083-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task1JRg7N/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1781955746769051-7083-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task1JRg7N/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1781955746769051-7083-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:36507
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-task1JRg7N/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1781955746769051-7083-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:38899
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:36507
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=required
--rpc_encryption=required
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-task1JRg7N/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1781955746769051-7083-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task1JRg7N/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1781955746769051-7083-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task1JRg7N/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1781955746769051-7083-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 11:46:22.340137 27239 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 11:46:22.340337 27239 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 11:46:22.340359 27239 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 11:46:22.341421 27239 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 11:46:22.341456 27239 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 11:46:22.341470 27239 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 11:46:22.341482 27239 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 11:46:22.341495 27239 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 11:46:22.342897 27239 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:38899
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task1JRg7N/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1781955746769051-7083-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task1JRg7N/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1781955746769051-7083-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:36507
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task1JRg7N/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1781955746769051-7083-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:36507
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task1JRg7N/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1781955746769051-7083-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.27239
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task1JRg7N/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1781955746769051-7083-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 20 Jun 2026 10:32:37 UTC on ccycloud.quasar-bgyofv.root.comops.site
build id 4457173
I20260620 11:46:22.343081 27239 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 11:46:22.343247 27239 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 11:46:22.347909 27244 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 11:46:22.347921 27245 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 11:46:22.347934 27247 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 11:46:22.348177 27239 server_base.cc:1034] running on GCE node
I20260620 11:46:22.348418 27239 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 11:46:22.348685 27239 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 11:46:22.349880 27239 hybrid_clock.cc:648] HybridClock initialized: now 1781955982349872 us; error 36 us; skew 500 ppm
Jun 20 11:46:22 dist-test-slave-cjc6 krb5kdc[27226](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781955982, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 11:46:22.352466 27239 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260620 11:46:22.352933 27239 webserver.cc:460] Webserver started at http://127.0.0.1:42593/ using document root <none> and password file <none>
I20260620 11:46:22.353193 27239 fs_manager.cc:359] Metadata directory not provided
I20260620 11:46:22.353271 27239 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 11:46:22.353426 27239 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 11:46:22.354581 27239 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task1JRg7N/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1781955746769051-7083-0/minicluster-data/master-0/data/instance:
uuid: "178fbb6338a54052954896ad2de3cd1a"
format_stamp: "Formatted at 2026-06-20 11:46:22 on dist-test-slave-cjc6"
server_key: "939085c3baeec4cd55844ce1ad5d1c26"
server_key_iv: "2aef7e5413c7155daad58cc6e4446c94"
server_key_version: "encryptionkey@0"
I20260620 11:46:22.355046 27239 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task1JRg7N/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1781955746769051-7083-0/minicluster-data/master-0/wal/instance:
uuid: "178fbb6338a54052954896ad2de3cd1a"
format_stamp: "Formatted at 2026-06-20 11:46:22 on dist-test-slave-cjc6"
server_key: "939085c3baeec4cd55844ce1ad5d1c26"
server_key_iv: "2aef7e5413c7155daad58cc6e4446c94"
server_key_version: "encryptionkey@0"
I20260620 11:46:22.356429 27239 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.002s
I20260620 11:46:22.357182 27254 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 11:46:22.357355 27239 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.001s
I20260620 11:46:22.357470 27239 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task1JRg7N/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1781955746769051-7083-0/minicluster-data/master-0/data,/tmp/dist-test-task1JRg7N/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1781955746769051-7083-0/minicluster-data/master-0/wal
uuid: "178fbb6338a54052954896ad2de3cd1a"
format_stamp: "Formatted at 2026-06-20 11:46:22 on dist-test-slave-cjc6"
server_key: "939085c3baeec4cd55844ce1ad5d1c26"
server_key_iv: "2aef7e5413c7155daad58cc6e4446c94"
server_key_version: "encryptionkey@0"
I20260620 11:46:22.357548 27239 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task1JRg7N/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1781955746769051-7083-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task1JRg7N/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1781955746769051-7083-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task1JRg7N/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1781955746769051-7083-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 11:46:22.379513 27239 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 11:46:22.381533 27239 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 11:46:22.381772 27239 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 11:46:22.386245 27239 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:36507
I20260620 11:46:22.386281 27306 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:36507 every 8 connection(s)
I20260620 11:46:22.386682 27239 server_base.cc:1166] Dumped server information to /tmp/dist-test-task1JRg7N/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1781955746769051-7083-0/minicluster-data/master-0/data/info.pb
I20260620 11:46:22.387679 27307 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 11:46:22.390014 27307 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 178fbb6338a54052954896ad2de3cd1a: Bootstrap starting.
I20260620 11:46:22.390584 27307 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 178fbb6338a54052954896ad2de3cd1a: Neither blocks nor log segments found. Creating new log.
I20260620 11:46:22.390884 27307 log.cc:824] T 00000000000000000000000000000000 P 178fbb6338a54052954896ad2de3cd1a: Log is configured to *not* fsync() on all Append() calls
I20260620 11:46:22.391571 27307 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 178fbb6338a54052954896ad2de3cd1a: No bootstrap required, opened a new log
I20260620 11:46:22.393137 27307 raft_consensus.cc:348] T 00000000000000000000000000000000 P 178fbb6338a54052954896ad2de3cd1a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "178fbb6338a54052954896ad2de3cd1a" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 36507 } }
I20260620 11:46:22.393275 27307 raft_consensus.cc:374] T 00000000000000000000000000000000 P 178fbb6338a54052954896ad2de3cd1a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 11:46:22.393296 27307 raft_consensus.cc:729] T 00000000000000000000000000000000 P 178fbb6338a54052954896ad2de3cd1a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 178fbb6338a54052954896ad2de3cd1a, State: Initialized, Role: FOLLOWER
I20260620 11:46:22.393446 27307 consensus_queue.cc:260] T 00000000000000000000000000000000 P 178fbb6338a54052954896ad2de3cd1a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "178fbb6338a54052954896ad2de3cd1a" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 36507 } }
I20260620 11:46:22.393527 27307 raft_consensus.cc:388] T 00000000000000000000000000000000 P 178fbb6338a54052954896ad2de3cd1a [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260620 11:46:22.393548 27307 raft_consensus.cc:482] T 00000000000000000000000000000000 P 178fbb6338a54052954896ad2de3cd1a [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260620 11:46:22.393595 27307 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 178fbb6338a54052954896ad2de3cd1a [term 0 FOLLOWER]: Advancing to term 1
I20260620 11:46:22.394124 27307 raft_consensus.cc:504] T 00000000000000000000000000000000 P 178fbb6338a54052954896ad2de3cd1a [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "178fbb6338a54052954896ad2de3cd1a" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 36507 } }
I20260620 11:46:22.394234 27307 leader_election.cc:302] T 00000000000000000000000000000000 P 178fbb6338a54052954896ad2de3cd1a [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 178fbb6338a54052954896ad2de3cd1a; no voters:
I20260620 11:46:22.394414 27307 leader_election.cc:288] T 00000000000000000000000000000000 P 178fbb6338a54052954896ad2de3cd1a [CANDIDATE]: Term 1 election: Requested vote from peers
I20260620 11:46:22.394475 27310 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 178fbb6338a54052954896ad2de3cd1a [term 1 FOLLOWER]: Leader election won for term 1
I20260620 11:46:22.394650 27310 raft_consensus.cc:686] T 00000000000000000000000000000000 P 178fbb6338a54052954896ad2de3cd1a [term 1 LEADER]: Becoming Leader. State: Replica: 178fbb6338a54052954896ad2de3cd1a, State: Running, Role: LEADER
I20260620 11:46:22.394699 27307 sys_catalog.cc:564] T 00000000000000000000000000000000 P 178fbb6338a54052954896ad2de3cd1a [sys.catalog]: configured and running, proceeding with master startup.
I20260620 11:46:22.394740 27310 consensus_queue.cc:237] T 00000000000000000000000000000000 P 178fbb6338a54052954896ad2de3cd1a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "178fbb6338a54052954896ad2de3cd1a" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 36507 } }
I20260620 11:46:22.395177 27311 sys_catalog.cc:455] T 00000000000000000000000000000000 P 178fbb6338a54052954896ad2de3cd1a [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "178fbb6338a54052954896ad2de3cd1a" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "178fbb6338a54052954896ad2de3cd1a" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 36507 } } }
I20260620 11:46:22.395268 27311 sys_catalog.cc:458] T 00000000000000000000000000000000 P 178fbb6338a54052954896ad2de3cd1a [sys.catalog]: This master's current role is: LEADER
I20260620 11:46:22.395428 27312 sys_catalog.cc:455] T 00000000000000000000000000000000 P 178fbb6338a54052954896ad2de3cd1a [sys.catalog]: SysCatalogTable state changed. Reason: New leader 178fbb6338a54052954896ad2de3cd1a. Latest consensus state: current_term: 1 leader_uuid: "178fbb6338a54052954896ad2de3cd1a" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "178fbb6338a54052954896ad2de3cd1a" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 36507 } } }
I20260620 11:46:22.395576 27312 sys_catalog.cc:458] T 00000000000000000000000000000000 P 178fbb6338a54052954896ad2de3cd1a [sys.catalog]: This master's current role is: LEADER
I20260620 11:46:22.395601 7083 external_mini_cluster.cc:1388] Started /tmp/dist-test-task1JRg7N/build/release/bin/kudu as pid 27239
I20260620 11:46:22.395730 7083 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task1JRg7N/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1781955746769051-7083-0/minicluster-data/master-0/wal/instance
I20260620 11:46:22.395910 27319 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260620 11:46:22.395951 7083 external_mini_cluster.cc:1428] Setting key b9baafe990c4eee77fae66cb8777360c
I20260620 11:46:22.396489 27319 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260620 11:46:22.397922 27319 catalog_manager.cc:1347] Generated new cluster ID: 09cf5ee24581436a8400bb3893647ce3
I20260620 11:46:22.397976 27319 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
Jun 20 11:46:22 dist-test-slave-cjc6 krb5kdc[27226](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781955982, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.0.1@KRBTEST.COM
I20260620 11:46:22.405387 27319 catalog_manager.cc:1370] Generated new certificate authority record
I20260620 11:46:22.406090 27319 catalog_manager.cc:1504] Loading token signing keys...
I20260620 11:46:22.407045 27328 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 11:46:22.396704 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41965 (local address 127.0.0.1:36507)
0620 11:46:22.396943 (+ 239us) server_negotiation.cc:206] Beginning negotiation
0620 11:46:22.396952 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0620 11:46:22.397123 (+ 171us) server_negotiation.cc:407] Connection header received
0620 11:46:22.397278 (+ 155us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 11:46:22.397287 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 11:46:22.397355 (+ 68us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 11:46:22.397502 (+ 147us) server_negotiation.cc:226] Negotiated authn=SASL
0620 11:46:22.398382 (+ 880us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 11:46:22.399295 (+ 913us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 11:46:22.400067 (+ 772us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 11:46:22.400488 (+ 421us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 11:46:22.400603 (+ 115us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 11:46:22.403015 (+ 2412us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 11:46:22.403019 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 11:46:22.403019 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 11:46:22.403038 (+ 19us) server_negotiation.cc:922] calling sasl_server_start()
0620 11:46:22.405329 (+ 2291us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 11:46:22.405806 (+ 477us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 11:46:22.405809 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 11:46:22.405810 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 11:46:22.405907 (+ 97us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 11:46:22.406197 (+ 290us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 11:46:22.406198 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 11:46:22.406199 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 11:46:22.406435 (+ 236us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 11:46:22.406636 (+ 201us) server_negotiation.cc:1035] Waiting for connection context
0620 11:46:22.406734 (+ 98us) server_negotiation.cc:299] Negotiation successful
0620 11:46:22.406846 (+ 112us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":189,"thread_start_us":60,"threads_started":1}
I20260620 11:46:22.408062 7083 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260620 11:46:22.412420 27319 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 178fbb6338a54052954896ad2de3cd1a: Generated new TSK 0
I20260620 11:46:22.412593 27319 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260620 11:46:22.415951 27328 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 11:46:22.409412 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45015 (local address 127.0.0.1:36507)
0620 11:46:22.409451 (+ 39us) server_negotiation.cc:206] Beginning negotiation
0620 11:46:22.409453 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 11:46:22.409869 (+ 416us) server_negotiation.cc:407] Connection header received
0620 11:46:22.409959 (+ 90us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 11:46:22.409960 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 11:46:22.409971 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 11:46:22.410020 (+ 49us) server_negotiation.cc:226] Negotiated authn=SASL
0620 11:46:22.410524 (+ 504us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 11:46:22.411043 (+ 519us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 11:46:22.411766 (+ 723us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 11:46:22.411999 (+ 233us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 11:46:22.412059 (+ 60us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 11:46:22.413398 (+ 1339us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 11:46:22.413400 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 11:46:22.413400 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 11:46:22.413407 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0620 11:46:22.414826 (+ 1419us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 11:46:22.415182 (+ 356us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 11:46:22.415184 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 11:46:22.415184 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 11:46:22.415241 (+ 57us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 11:46:22.415464 (+ 223us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 11:46:22.415465 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 11:46:22.415465 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 11:46:22.415567 (+ 102us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 11:46:22.415675 (+ 108us) server_negotiation.cc:1035] Waiting for connection context
0620 11:46:22.415796 (+ 121us) server_negotiation.cc:299] Negotiation successful
0620 11:46:22.415857 (+ 61us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":18}
I20260620 11:46:22.418427 7083 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task1JRg7N/build/release/bin/kudu with pid 27239
W20260620 11:46:22.422289 27224 connection.cc:535] client connection to 127.0.0.1:36507 recv error: Network error: failed to read from TLS socket (remote: 127.0.0.1:36507): Connection reset by peer (error 104)
2026-06-20T11:46:22Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/0 (1307 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/1
Loading random data
Initializing database '/tmp/dist-test-task1JRg7N/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1781955746769051-7083-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 20 11:46:22 dist-test-slave-cjc6 krb5kdc[27338](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 20 11:46:22 dist-test-slave-cjc6 krb5kdc[27338](info): set up 2 sockets
Jun 20 11:46:22 dist-test-slave-cjc6 krb5kdc[27338](info): commencing operation
krb5kdc: starting...
W20260620 11:46:23.457854 7083 mini_kdc.cc:121] Time spent starting KDC: real 1.022s user 0.000s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task1JRg7N/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1781955746769051-7083-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task1JRg7N/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1781955746769051-7083-0/krb5kdc/test-user.keytab.
Jun 20 11:46:23 dist-test-slave-cjc6 krb5kdc[27338](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781955983, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-20T11:46:23Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-20T11:46:23Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task1JRg7N/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1781955746769051-7083-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task1JRg7N/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1781955746769051-7083-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260620 11:46:23.572943 7083 external_mini_cluster.cc:1326] Running /tmp/dist-test-task1JRg7N/build/release/bin/kudu
/tmp/dist-test-task1JRg7N/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task1JRg7N/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1781955746769051-7083-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task1JRg7N/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1781955746769051-7083-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task1JRg7N/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1781955746769051-7083-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task1JRg7N/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1781955746769051-7083-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:32939
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-task1JRg7N/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1781955746769051-7083-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:36851
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:32939
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=required
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-task1JRg7N/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1781955746769051-7083-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task1JRg7N/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1781955746769051-7083-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task1JRg7N/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1781955746769051-7083-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 11:46:23.652122 27351 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 11:46:23.652354 27351 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 11:46:23.652391 27351 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 11:46:23.653513 27351 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 11:46:23.653589 27351 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 11:46:23.653612 27351 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 11:46:23.653635 27351 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 11:46:23.653661 27351 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 11:46:23.655138 27351 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:36851
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task1JRg7N/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1781955746769051-7083-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task1JRg7N/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1781955746769051-7083-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:32939
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task1JRg7N/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1781955746769051-7083-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:32939
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task1JRg7N/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1781955746769051-7083-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.27351
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task1JRg7N/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1781955746769051-7083-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 20 Jun 2026 10:32:37 UTC on ccycloud.quasar-bgyofv.root.comops.site
build id 4457173
I20260620 11:46:23.655404 27351 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 11:46:23.655627 27351 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 11:46:23.660426 27357 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 11:46:23.660444 27359 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 11:46:23.660480 27356 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 11:46:23.660789 27351 server_base.cc:1034] running on GCE node
I20260620 11:46:23.661033 27351 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 11:46:23.661263 27351 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 11:46:23.662398 27351 hybrid_clock.cc:648] HybridClock initialized: now 1781955983662385 us; error 29 us; skew 500 ppm
Jun 20 11:46:23 dist-test-slave-cjc6 krb5kdc[27338](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781955983, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 11:46:23.664848 27351 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260620 11:46:23.665222 27351 webserver.cc:460] Webserver started at http://127.0.0.1:40049/ using document root <none> and password file <none>
I20260620 11:46:23.665441 27351 fs_manager.cc:359] Metadata directory not provided
I20260620 11:46:23.665483 27351 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 11:46:23.665570 27351 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 11:46:23.666625 27351 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task1JRg7N/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1781955746769051-7083-0/minicluster-data/master-0/data/instance:
uuid: "6324a9bad8f74595877f9e3239394f57"
format_stamp: "Formatted at 2026-06-20 11:46:23 on dist-test-slave-cjc6"
server_key: "9ab7da039e4b7a45a8d41d28a982fa56"
server_key_iv: "af428de306051f14a327cf29ed18e08d"
server_key_version: "encryptionkey@0"
I20260620 11:46:23.666952 27351 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task1JRg7N/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1781955746769051-7083-0/minicluster-data/master-0/wal/instance:
uuid: "6324a9bad8f74595877f9e3239394f57"
format_stamp: "Formatted at 2026-06-20 11:46:23 on dist-test-slave-cjc6"
server_key: "9ab7da039e4b7a45a8d41d28a982fa56"
server_key_iv: "af428de306051f14a327cf29ed18e08d"
server_key_version: "encryptionkey@0"
I20260620 11:46:23.668241 27351 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260620 11:46:23.668937 27366 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 11:46:23.669106 27351 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260620 11:46:23.669188 27351 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task1JRg7N/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1781955746769051-7083-0/minicluster-data/master-0/data,/tmp/dist-test-task1JRg7N/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1781955746769051-7083-0/minicluster-data/master-0/wal
uuid: "6324a9bad8f74595877f9e3239394f57"
format_stamp: "Formatted at 2026-06-20 11:46:23 on dist-test-slave-cjc6"
server_key: "9ab7da039e4b7a45a8d41d28a982fa56"
server_key_iv: "af428de306051f14a327cf29ed18e08d"
server_key_version: "encryptionkey@0"
I20260620 11:46:23.669238 27351 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task1JRg7N/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1781955746769051-7083-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task1JRg7N/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1781955746769051-7083-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task1JRg7N/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1781955746769051-7083-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 11:46:23.676208 27351 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 11:46:23.678467 27351 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 11:46:23.678572 27351 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 11:46:23.682291 27351 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:32939
I20260620 11:46:23.682322 27418 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:32939 every 8 connection(s)
I20260620 11:46:23.682689 27351 server_base.cc:1166] Dumped server information to /tmp/dist-test-task1JRg7N/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1781955746769051-7083-0/minicluster-data/master-0/data/info.pb
I20260620 11:46:23.683650 27419 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 11:46:23.685938 27419 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 6324a9bad8f74595877f9e3239394f57: Bootstrap starting.
I20260620 11:46:23.686507 27419 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 6324a9bad8f74595877f9e3239394f57: Neither blocks nor log segments found. Creating new log.
I20260620 11:46:23.686789 27419 log.cc:824] T 00000000000000000000000000000000 P 6324a9bad8f74595877f9e3239394f57: Log is configured to *not* fsync() on all Append() calls
I20260620 11:46:23.687386 7083 external_mini_cluster.cc:1388] Started /tmp/dist-test-task1JRg7N/build/release/bin/kudu as pid 27351
I20260620 11:46:23.687515 27419 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 6324a9bad8f74595877f9e3239394f57: No bootstrap required, opened a new log
I20260620 11:46:23.687618 7083 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task1JRg7N/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1781955746769051-7083-0/minicluster-data/master-0/wal/instance
I20260620 11:46:23.687845 7083 external_mini_cluster.cc:1428] Setting key b09df029b461506f82fe370283a8d07c
I20260620 11:46:23.689404 27419 raft_consensus.cc:348] T 00000000000000000000000000000000 P 6324a9bad8f74595877f9e3239394f57 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6324a9bad8f74595877f9e3239394f57" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 32939 } }
I20260620 11:46:23.689567 27419 raft_consensus.cc:374] T 00000000000000000000000000000000 P 6324a9bad8f74595877f9e3239394f57 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 11:46:23.689606 27419 raft_consensus.cc:729] T 00000000000000000000000000000000 P 6324a9bad8f74595877f9e3239394f57 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6324a9bad8f74595877f9e3239394f57, State: Initialized, Role: FOLLOWER
I20260620 11:46:23.689759 27419 consensus_queue.cc:260] T 00000000000000000000000000000000 P 6324a9bad8f74595877f9e3239394f57 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6324a9bad8f74595877f9e3239394f57" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 32939 } }
I20260620 11:46:23.689858 27419 raft_consensus.cc:388] T 00000000000000000000000000000000 P 6324a9bad8f74595877f9e3239394f57 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260620 11:46:23.689883 27419 raft_consensus.cc:482] T 00000000000000000000000000000000 P 6324a9bad8f74595877f9e3239394f57 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260620 11:46:23.689953 27419 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 6324a9bad8f74595877f9e3239394f57 [term 0 FOLLOWER]: Advancing to term 1
I20260620 11:46:23.690477 27419 raft_consensus.cc:504] T 00000000000000000000000000000000 P 6324a9bad8f74595877f9e3239394f57 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6324a9bad8f74595877f9e3239394f57" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 32939 } }
I20260620 11:46:23.690559 27419 leader_election.cc:302] T 00000000000000000000000000000000 P 6324a9bad8f74595877f9e3239394f57 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 6324a9bad8f74595877f9e3239394f57; no voters:
I20260620 11:46:23.690747 27419 leader_election.cc:288] T 00000000000000000000000000000000 P 6324a9bad8f74595877f9e3239394f57 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260620 11:46:23.690771 27424 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 6324a9bad8f74595877f9e3239394f57 [term 1 FOLLOWER]: Leader election won for term 1
I20260620 11:46:23.690883 27424 raft_consensus.cc:686] T 00000000000000000000000000000000 P 6324a9bad8f74595877f9e3239394f57 [term 1 LEADER]: Becoming Leader. State: Replica: 6324a9bad8f74595877f9e3239394f57, State: Running, Role: LEADER
I20260620 11:46:23.690964 27424 consensus_queue.cc:237] T 00000000000000000000000000000000 P 6324a9bad8f74595877f9e3239394f57 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6324a9bad8f74595877f9e3239394f57" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 32939 } }
I20260620 11:46:23.690994 27419 sys_catalog.cc:564] T 00000000000000000000000000000000 P 6324a9bad8f74595877f9e3239394f57 [sys.catalog]: configured and running, proceeding with master startup.
I20260620 11:46:23.691298 27424 sys_catalog.cc:455] T 00000000000000000000000000000000 P 6324a9bad8f74595877f9e3239394f57 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 6324a9bad8f74595877f9e3239394f57. Latest consensus state: current_term: 1 leader_uuid: "6324a9bad8f74595877f9e3239394f57" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6324a9bad8f74595877f9e3239394f57" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 32939 } } }
I20260620 11:46:23.691376 27424 sys_catalog.cc:458] T 00000000000000000000000000000000 P 6324a9bad8f74595877f9e3239394f57 [sys.catalog]: This master's current role is: LEADER
I20260620 11:46:23.691313 27425 sys_catalog.cc:455] T 00000000000000000000000000000000 P 6324a9bad8f74595877f9e3239394f57 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "6324a9bad8f74595877f9e3239394f57" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6324a9bad8f74595877f9e3239394f57" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 32939 } } }
I20260620 11:46:23.691545 27425 sys_catalog.cc:458] T 00000000000000000000000000000000 P 6324a9bad8f74595877f9e3239394f57 [sys.catalog]: This master's current role is: LEADER
I20260620 11:46:23.691583 27429 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260620 11:46:23.692122 27429 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260620 11:46:23.694484 27423 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 11:46:23.688331 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48489 (local address 127.0.0.1:32939)
0620 11:46:23.688518 (+ 187us) server_negotiation.cc:206] Beginning negotiation
0620 11:46:23.688525 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0620 11:46:23.688550 (+ 25us) server_negotiation.cc:407] Connection header received
0620 11:46:23.689388 (+ 838us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 11:46:23.689394 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 11:46:23.689441 (+ 47us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 11:46:23.689549 (+ 108us) server_negotiation.cc:226] Negotiated authn=SASL
0620 11:46:23.689803 (+ 254us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 11:46:23.691276 (+ 1473us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 11:46:23.692325 (+ 1049us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 11:46:23.692919 (+ 594us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 11:46:23.693023 (+ 104us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 11:46:23.693792 (+ 769us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 11:46:23.693796 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 11:46:23.693797 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 11:46:23.694007 (+ 210us) server_negotiation.cc:922] calling sasl_server_start()
0620 11:46:23.694045 (+ 38us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 11:46:23.694062 (+ 17us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 11:46:23.694150 (+ 88us) server_negotiation.cc:1035] Waiting for connection context
0620 11:46:23.694247 (+ 97us) server_negotiation.cc:299] Negotiation successful
0620 11:46:23.694273 (+ 26us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":132,"thread_start_us":94,"threads_started":1}
I20260620 11:46:23.694816 27429 catalog_manager.cc:1347] Generated new cluster ID: 8ba563717b22435f98f97f33367c442f
I20260620 11:46:23.694868 27429 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260620 11:46:23.695415 7083 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260620 11:46:23.700366 27423 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 11:46:23.696986 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59791 (local address 127.0.0.1:32939)
0620 11:46:23.697019 (+ 33us) server_negotiation.cc:206] Beginning negotiation
0620 11:46:23.697021 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 11:46:23.697041 (+ 20us) server_negotiation.cc:407] Connection header received
0620 11:46:23.697101 (+ 60us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 11:46:23.697102 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 11:46:23.697110 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 11:46:23.697148 (+ 38us) server_negotiation.cc:226] Negotiated authn=SASL
0620 11:46:23.697296 (+ 148us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 11:46:23.698010 (+ 714us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 11:46:23.699026 (+ 1016us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 11:46:23.699335 (+ 309us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 11:46:23.699394 (+ 59us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 11:46:23.700105 (+ 711us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 11:46:23.700106 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 11:46:23.700107 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 11:46:23.700111 (+ 4us) server_negotiation.cc:922] calling sasl_server_start()
0620 11:46:23.700125 (+ 14us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 11:46:23.700129 (+ 4us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 11:46:23.700183 (+ 54us) server_negotiation.cc:1035] Waiting for connection context
0620 11:46:23.700284 (+ 101us) server_negotiation.cc:299] Negotiation successful
0620 11:46:23.700295 (+ 11us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":17}
I20260620 11:46:23.717459 27429 catalog_manager.cc:1370] Generated new certificate authority record
I20260620 11:46:23.718047 27429 catalog_manager.cc:1504] Loading token signing keys...
I20260620 11:46:23.722623 27429 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 6324a9bad8f74595877f9e3239394f57: Generated new TSK 0
I20260620 11:46:23.722813 27429 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260620 11:46:23.729470 7083 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task1JRg7N/build/release/bin/kudu with pid 27351
W20260620 11:46:23.733855 27336 connection.cc:535] client connection to 127.0.0.1:32939 recv error: Network error: failed to read from TLS socket (remote: 127.0.0.1:32939): Connection reset by peer (error 104)
2026-06-20T11:46:23Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/1 (1311 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/2
Loading random data
Initializing database '/tmp/dist-test-task1JRg7N/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1781955746769051-7083-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 20 11:46:23 dist-test-slave-cjc6 krb5kdc[27450](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 20 11:46:23 dist-test-slave-cjc6 krb5kdc[27450](info): set up 2 sockets
Jun 20 11:46:23 dist-test-slave-cjc6 krb5kdc[27450](info): commencing operation
krb5kdc: starting...
W20260620 11:46:24.761828 7083 mini_kdc.cc:121] Time spent starting KDC: real 1.014s user 0.002s sys 0.004s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task1JRg7N/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1781955746769051-7083-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task1JRg7N/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1781955746769051-7083-0/krb5kdc/test-user.keytab.
Jun 20 11:46:24 dist-test-slave-cjc6 krb5kdc[27450](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781955984, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-20T11:46:24Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-20T11:46:24Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task1JRg7N/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1781955746769051-7083-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task1JRg7N/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1781955746769051-7083-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260620 11:46:24.876336 7083 external_mini_cluster.cc:1326] Running /tmp/dist-test-task1JRg7N/build/release/bin/kudu
/tmp/dist-test-task1JRg7N/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task1JRg7N/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1781955746769051-7083-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task1JRg7N/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1781955746769051-7083-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task1JRg7N/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1781955746769051-7083-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task1JRg7N/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1781955746769051-7083-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:41375
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-task1JRg7N/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1781955746769051-7083-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:45813
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:41375
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=disabled
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-task1JRg7N/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1781955746769051-7083-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task1JRg7N/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1781955746769051-7083-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task1JRg7N/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1781955746769051-7083-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 11:46:24.956542 27463 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 11:46:24.956810 27463 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 11:46:24.956853 27463 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 11:46:24.958058 27463 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 11:46:24.958128 27463 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 11:46:24.958153 27463 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 11:46:24.958184 27463 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 11:46:24.958214 27463 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 11:46:24.959677 27463 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:45813
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task1JRg7N/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1781955746769051-7083-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task1JRg7N/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1781955746769051-7083-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:41375
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task1JRg7N/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1781955746769051-7083-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:41375
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task1JRg7N/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1781955746769051-7083-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.27463
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task1JRg7N/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1781955746769051-7083-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 20 Jun 2026 10:32:37 UTC on ccycloud.quasar-bgyofv.root.comops.site
build id 4457173
I20260620 11:46:24.959954 27463 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 11:46:24.960178 27463 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 11:46:24.965010 27471 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 11:46:24.965111 27469 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 11:46:24.964973 27468 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 11:46:24.965242 27463 server_base.cc:1034] running on GCE node
I20260620 11:46:24.965514 27463 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 11:46:24.965906 27463 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 11:46:24.967101 27463 hybrid_clock.cc:648] HybridClock initialized: now 1781955984967091 us; error 39 us; skew 500 ppm
Jun 20 11:46:24 dist-test-slave-cjc6 krb5kdc[27450](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781955984, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 11:46:24.969801 27463 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260620 11:46:24.970230 27463 webserver.cc:460] Webserver started at http://127.0.0.1:46795/ using document root <none> and password file <none>
I20260620 11:46:24.970494 27463 fs_manager.cc:359] Metadata directory not provided
I20260620 11:46:24.970559 27463 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 11:46:24.970710 27463 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 11:46:24.971769 27463 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task1JRg7N/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1781955746769051-7083-0/minicluster-data/master-0/data/instance:
uuid: "3065f5a4b93d4ebcaf8cdf7629e16359"
format_stamp: "Formatted at 2026-06-20 11:46:24 on dist-test-slave-cjc6"
server_key: "a7677790e8fa22f5d6f7a1ed56e40ef1"
server_key_iv: "09d76a3da78abc649b1386774b66a842"
server_key_version: "encryptionkey@0"
I20260620 11:46:24.972131 27463 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task1JRg7N/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1781955746769051-7083-0/minicluster-data/master-0/wal/instance:
uuid: "3065f5a4b93d4ebcaf8cdf7629e16359"
format_stamp: "Formatted at 2026-06-20 11:46:24 on dist-test-slave-cjc6"
server_key: "a7677790e8fa22f5d6f7a1ed56e40ef1"
server_key_iv: "09d76a3da78abc649b1386774b66a842"
server_key_version: "encryptionkey@0"
I20260620 11:46:24.973455 27463 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260620 11:46:24.974360 27478 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 11:46:24.974551 27463 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.001s
I20260620 11:46:24.974651 27463 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task1JRg7N/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1781955746769051-7083-0/minicluster-data/master-0/data,/tmp/dist-test-task1JRg7N/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1781955746769051-7083-0/minicluster-data/master-0/wal
uuid: "3065f5a4b93d4ebcaf8cdf7629e16359"
format_stamp: "Formatted at 2026-06-20 11:46:24 on dist-test-slave-cjc6"
server_key: "a7677790e8fa22f5d6f7a1ed56e40ef1"
server_key_iv: "09d76a3da78abc649b1386774b66a842"
server_key_version: "encryptionkey@0"
I20260620 11:46:24.974740 27463 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task1JRg7N/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1781955746769051-7083-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task1JRg7N/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1781955746769051-7083-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task1JRg7N/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1781955746769051-7083-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 11:46:24.978060 27463 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 11:46:24.979892 27463 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 11:46:24.980052 27463 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 11:46:24.984016 27463 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:41375
I20260620 11:46:24.984048 27530 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:41375 every 8 connection(s)
I20260620 11:46:24.984385 27463 server_base.cc:1166] Dumped server information to /tmp/dist-test-task1JRg7N/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1781955746769051-7083-0/minicluster-data/master-0/data/info.pb
I20260620 11:46:24.985379 27531 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 11:46:24.987722 27531 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 3065f5a4b93d4ebcaf8cdf7629e16359: Bootstrap starting.
I20260620 11:46:24.988305 27531 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 3065f5a4b93d4ebcaf8cdf7629e16359: Neither blocks nor log segments found. Creating new log.
I20260620 11:46:24.988603 27531 log.cc:824] T 00000000000000000000000000000000 P 3065f5a4b93d4ebcaf8cdf7629e16359: Log is configured to *not* fsync() on all Append() calls
I20260620 11:46:24.989348 27531 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 3065f5a4b93d4ebcaf8cdf7629e16359: No bootstrap required, opened a new log
I20260620 11:46:24.990926 7083 external_mini_cluster.cc:1388] Started /tmp/dist-test-task1JRg7N/build/release/bin/kudu as pid 27463
I20260620 11:46:24.991084 7083 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task1JRg7N/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1781955746769051-7083-0/minicluster-data/master-0/wal/instance
I20260620 11:46:24.991007 27531 raft_consensus.cc:348] T 00000000000000000000000000000000 P 3065f5a4b93d4ebcaf8cdf7629e16359 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3065f5a4b93d4ebcaf8cdf7629e16359" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 41375 } }
I20260620 11:46:24.991231 27531 raft_consensus.cc:374] T 00000000000000000000000000000000 P 3065f5a4b93d4ebcaf8cdf7629e16359 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 11:46:24.991276 27531 raft_consensus.cc:729] T 00000000000000000000000000000000 P 3065f5a4b93d4ebcaf8cdf7629e16359 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3065f5a4b93d4ebcaf8cdf7629e16359, State: Initialized, Role: FOLLOWER
I20260620 11:46:24.991324 7083 external_mini_cluster.cc:1428] Setting key 8d4d5dbac2d008dffcdd8bc77cce24db
I20260620 11:46:24.991426 27531 consensus_queue.cc:260] T 00000000000000000000000000000000 P 3065f5a4b93d4ebcaf8cdf7629e16359 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3065f5a4b93d4ebcaf8cdf7629e16359" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 41375 } }
I20260620 11:46:24.991514 27531 raft_consensus.cc:388] T 00000000000000000000000000000000 P 3065f5a4b93d4ebcaf8cdf7629e16359 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260620 11:46:24.991559 27531 raft_consensus.cc:482] T 00000000000000000000000000000000 P 3065f5a4b93d4ebcaf8cdf7629e16359 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260620 11:46:24.991611 27531 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 3065f5a4b93d4ebcaf8cdf7629e16359 [term 0 FOLLOWER]: Advancing to term 1
I20260620 11:46:24.992254 27531 raft_consensus.cc:504] T 00000000000000000000000000000000 P 3065f5a4b93d4ebcaf8cdf7629e16359 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3065f5a4b93d4ebcaf8cdf7629e16359" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 41375 } }
I20260620 11:46:24.992375 27531 leader_election.cc:302] T 00000000000000000000000000000000 P 3065f5a4b93d4ebcaf8cdf7629e16359 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 3065f5a4b93d4ebcaf8cdf7629e16359; no voters:
I20260620 11:46:24.992574 27531 leader_election.cc:288] T 00000000000000000000000000000000 P 3065f5a4b93d4ebcaf8cdf7629e16359 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260620 11:46:24.992606 27535 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 3065f5a4b93d4ebcaf8cdf7629e16359 [term 1 FOLLOWER]: Leader election won for term 1
I20260620 11:46:24.992750 27535 raft_consensus.cc:686] T 00000000000000000000000000000000 P 3065f5a4b93d4ebcaf8cdf7629e16359 [term 1 LEADER]: Becoming Leader. State: Replica: 3065f5a4b93d4ebcaf8cdf7629e16359, State: Running, Role: LEADER
I20260620 11:46:24.992900 27531 sys_catalog.cc:564] T 00000000000000000000000000000000 P 3065f5a4b93d4ebcaf8cdf7629e16359 [sys.catalog]: configured and running, proceeding with master startup.
I20260620 11:46:24.992887 27535 consensus_queue.cc:237] T 00000000000000000000000000000000 P 3065f5a4b93d4ebcaf8cdf7629e16359 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3065f5a4b93d4ebcaf8cdf7629e16359" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 41375 } }
I20260620 11:46:24.994279 27537 sys_catalog.cc:455] T 00000000000000000000000000000000 P 3065f5a4b93d4ebcaf8cdf7629e16359 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 3065f5a4b93d4ebcaf8cdf7629e16359. Latest consensus state: current_term: 1 leader_uuid: "3065f5a4b93d4ebcaf8cdf7629e16359" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3065f5a4b93d4ebcaf8cdf7629e16359" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 41375 } } }
I20260620 11:46:24.994534 27538 sys_catalog.cc:455] T 00000000000000000000000000000000 P 3065f5a4b93d4ebcaf8cdf7629e16359 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "3065f5a4b93d4ebcaf8cdf7629e16359" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3065f5a4b93d4ebcaf8cdf7629e16359" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 41375 } } }
I20260620 11:46:24.994570 27537 sys_catalog.cc:458] T 00000000000000000000000000000000 P 3065f5a4b93d4ebcaf8cdf7629e16359 [sys.catalog]: This master's current role is: LEADER
I20260620 11:46:24.994596 27538 sys_catalog.cc:458] T 00000000000000000000000000000000 P 3065f5a4b93d4ebcaf8cdf7629e16359 [sys.catalog]: This master's current role is: LEADER
I20260620 11:46:24.994374 27536 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 11:46:24.992458 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45915 (local address 127.0.0.1:41375)
0620 11:46:24.992652 (+ 194us) server_negotiation.cc:206] Beginning negotiation
0620 11:46:24.992661 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0620 11:46:24.992695 (+ 34us) server_negotiation.cc:407] Connection header received
0620 11:46:24.992835 (+ 140us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 11:46:24.992842 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 11:46:24.992872 (+ 30us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 11:46:24.993033 (+ 161us) server_negotiation.cc:226] Negotiated authn=SASL
0620 11:46:24.993806 (+ 773us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 11:46:24.993810 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 11:46:24.993810 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 11:46:24.993814 (+ 4us) server_negotiation.cc:922] calling sasl_server_start()
0620 11:46:24.993862 (+ 48us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 11:46:24.993877 (+ 15us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 11:46:24.993951 (+ 74us) server_negotiation.cc:1035] Waiting for connection context
0620 11:46:24.994092 (+ 141us) server_negotiation.cc:299] Negotiation successful
0620 11:46:24.994136 (+ 44us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":137,"thread_start_us":109,"threads_started":1}
W20260620 11:46:24.994771 27550 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 3065f5a4b93d4ebcaf8cdf7629e16359: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260620 11:46:24.994820 27550 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260620 11:46:24.994889 27552 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260620 11:46:24.995056 27552 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260620 11:46:24.995750 7083 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260620 11:46:24.997963 27552 catalog_manager.cc:1347] Generated new cluster ID: b8b90fa3f5d84bf18ba78f732345e4bb
I20260620 11:46:24.998015 27552 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260620 11:46:25.011026 27536 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 11:46:24.997236 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48053 (local address 127.0.0.1:41375)
0620 11:46:24.997279 (+ 43us) server_negotiation.cc:206] Beginning negotiation
0620 11:46:24.997281 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 11:46:25.009937 (+ 12656us) server_negotiation.cc:407] Connection header received
0620 11:46:25.009999 (+ 62us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 11:46:25.010000 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 11:46:25.010008 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 11:46:25.010065 (+ 57us) server_negotiation.cc:226] Negotiated authn=SASL
0620 11:46:25.010780 (+ 715us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 11:46:25.010781 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 11:46:25.010782 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 11:46:25.010786 (+ 4us) server_negotiation.cc:922] calling sasl_server_start()
0620 11:46:25.010803 (+ 17us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 11:46:25.010807 (+ 4us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 11:46:25.010860 (+ 53us) server_negotiation.cc:1035] Waiting for connection context
0620 11:46:25.010928 (+ 68us) server_negotiation.cc:299] Negotiation successful
0620 11:46:25.010941 (+ 13us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":23}
I20260620 11:46:25.018970 27552 catalog_manager.cc:1370] Generated new certificate authority record
I20260620 11:46:25.019147 27552 catalog_manager.cc:1504] Loading token signing keys...
I20260620 11:46:25.031777 27552 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 3065f5a4b93d4ebcaf8cdf7629e16359: Generated new TSK 0
I20260620 11:46:25.032017 27552 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260620 11:46:25.036288 7083 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task1JRg7N/build/release/bin/kudu with pid 27463
2026-06-20T11:46:25Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/2 (1306 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/3
Loading random data
Initializing database '/tmp/dist-test-task1JRg7N/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1781955746769051-7083-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 20 11:46:25 dist-test-slave-cjc6 krb5kdc[27562](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 20 11:46:25 dist-test-slave-cjc6 krb5kdc[27562](info): set up 2 sockets
Jun 20 11:46:25 dist-test-slave-cjc6 krb5kdc[27562](info): commencing operation
krb5kdc: starting...
W20260620 11:46:26.069986 7083 mini_kdc.cc:121] Time spent starting KDC: real 1.016s user 0.000s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task1JRg7N/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1781955746769051-7083-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task1JRg7N/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1781955746769051-7083-0/krb5kdc/test-user.keytab.
Jun 20 11:46:26 dist-test-slave-cjc6 krb5kdc[27562](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781955986, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-20T11:46:26Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-20T11:46:26Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task1JRg7N/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1781955746769051-7083-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task1JRg7N/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1781955746769051-7083-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260620 11:46:26.184643 7083 external_mini_cluster.cc:1326] Running /tmp/dist-test-task1JRg7N/build/release/bin/kudu
/tmp/dist-test-task1JRg7N/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task1JRg7N/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1781955746769051-7083-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task1JRg7N/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1781955746769051-7083-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task1JRg7N/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1781955746769051-7083-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task1JRg7N/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1781955746769051-7083-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:44283
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-task1JRg7N/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1781955746769051-7083-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:40337
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:44283
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=required
--rpc_encryption=required
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-task1JRg7N/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1781955746769051-7083-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task1JRg7N/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1781955746769051-7083-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task1JRg7N/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1781955746769051-7083-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 11:46:26.264762 27575 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 11:46:26.265041 27575 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 11:46:26.265084 27575 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 11:46:26.266276 27575 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 11:46:26.266345 27575 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 11:46:26.266368 27575 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 11:46:26.266389 27575 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 11:46:26.266420 27575 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 11:46:26.268492 27575 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:40337
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task1JRg7N/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1781955746769051-7083-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task1JRg7N/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1781955746769051-7083-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:44283
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task1JRg7N/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1781955746769051-7083-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:44283
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task1JRg7N/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1781955746769051-7083-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.27575
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task1JRg7N/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1781955746769051-7083-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 20 Jun 2026 10:32:37 UTC on ccycloud.quasar-bgyofv.root.comops.site
build id 4457173
I20260620 11:46:26.268760 27575 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 11:46:26.269003 27575 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 11:46:26.273638 27583 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 11:46:26.273666 27575 server_base.cc:1034] running on GCE node
W20260620 11:46:26.273660 27581 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 11:46:26.273638 27580 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 11:46:26.274097 27575 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 11:46:26.274382 27575 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 11:46:26.275555 27575 hybrid_clock.cc:648] HybridClock initialized: now 1781955986275531 us; error 54 us; skew 500 ppm
Jun 20 11:46:26 dist-test-slave-cjc6 krb5kdc[27562](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781955986, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 11:46:26.283605 27575 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260620 11:46:26.284133 27575 webserver.cc:460] Webserver started at http://127.0.0.1:42007/ using document root <none> and password file <none>
I20260620 11:46:26.284368 27575 fs_manager.cc:359] Metadata directory not provided
I20260620 11:46:26.284438 27575 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 11:46:26.284591 27575 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 11:46:26.285648 27575 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task1JRg7N/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1781955746769051-7083-0/minicluster-data/master-0/data/instance:
uuid: "36e2a5b0519c4f7a892877635a453ce4"
format_stamp: "Formatted at 2026-06-20 11:46:26 on dist-test-slave-cjc6"
server_key: "3239a5044d1befbe3a8e3410c4243467"
server_key_iv: "34193babf83b705362ff1878518168f3"
server_key_version: "encryptionkey@0"
I20260620 11:46:26.286079 27575 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task1JRg7N/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1781955746769051-7083-0/minicluster-data/master-0/wal/instance:
uuid: "36e2a5b0519c4f7a892877635a453ce4"
format_stamp: "Formatted at 2026-06-20 11:46:26 on dist-test-slave-cjc6"
server_key: "3239a5044d1befbe3a8e3410c4243467"
server_key_iv: "34193babf83b705362ff1878518168f3"
server_key_version: "encryptionkey@0"
I20260620 11:46:26.287436 27575 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.001s sys 0.000s
I20260620 11:46:26.288221 27590 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 11:46:26.288403 27575 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260620 11:46:26.288509 27575 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task1JRg7N/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1781955746769051-7083-0/minicluster-data/master-0/data,/tmp/dist-test-task1JRg7N/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1781955746769051-7083-0/minicluster-data/master-0/wal
uuid: "36e2a5b0519c4f7a892877635a453ce4"
format_stamp: "Formatted at 2026-06-20 11:46:26 on dist-test-slave-cjc6"
server_key: "3239a5044d1befbe3a8e3410c4243467"
server_key_iv: "34193babf83b705362ff1878518168f3"
server_key_version: "encryptionkey@0"
I20260620 11:46:26.288581 27575 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task1JRg7N/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1781955746769051-7083-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task1JRg7N/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1781955746769051-7083-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task1JRg7N/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1781955746769051-7083-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 11:46:26.305440 27575 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 11:46:26.307451 27575 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 11:46:26.307622 27575 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 11:46:26.311683 27575 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:44283
I20260620 11:46:26.311719 27642 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:44283 every 8 connection(s)
I20260620 11:46:26.312065 27575 server_base.cc:1166] Dumped server information to /tmp/dist-test-task1JRg7N/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1781955746769051-7083-0/minicluster-data/master-0/data/info.pb
I20260620 11:46:26.313058 27643 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 11:46:26.315232 27643 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 36e2a5b0519c4f7a892877635a453ce4: Bootstrap starting.
I20260620 11:46:26.315758 27643 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 36e2a5b0519c4f7a892877635a453ce4: Neither blocks nor log segments found. Creating new log.
I20260620 11:46:26.315981 27643 log.cc:824] T 00000000000000000000000000000000 P 36e2a5b0519c4f7a892877635a453ce4: Log is configured to *not* fsync() on all Append() calls
I20260620 11:46:26.316645 27643 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 36e2a5b0519c4f7a892877635a453ce4: No bootstrap required, opened a new log
I20260620 11:46:26.318260 27643 raft_consensus.cc:348] T 00000000000000000000000000000000 P 36e2a5b0519c4f7a892877635a453ce4 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "36e2a5b0519c4f7a892877635a453ce4" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 44283 } }
I20260620 11:46:26.318401 27643 raft_consensus.cc:374] T 00000000000000000000000000000000 P 36e2a5b0519c4f7a892877635a453ce4 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 11:46:26.318424 27643 raft_consensus.cc:729] T 00000000000000000000000000000000 P 36e2a5b0519c4f7a892877635a453ce4 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 36e2a5b0519c4f7a892877635a453ce4, State: Initialized, Role: FOLLOWER
I20260620 11:46:26.318555 27643 consensus_queue.cc:260] T 00000000000000000000000000000000 P 36e2a5b0519c4f7a892877635a453ce4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "36e2a5b0519c4f7a892877635a453ce4" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 44283 } }
I20260620 11:46:26.318629 27643 raft_consensus.cc:388] T 00000000000000000000000000000000 P 36e2a5b0519c4f7a892877635a453ce4 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260620 11:46:26.318651 27643 raft_consensus.cc:482] T 00000000000000000000000000000000 P 36e2a5b0519c4f7a892877635a453ce4 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260620 11:46:26.318678 27643 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 36e2a5b0519c4f7a892877635a453ce4 [term 0 FOLLOWER]: Advancing to term 1
I20260620 11:46:26.319129 27643 raft_consensus.cc:504] T 00000000000000000000000000000000 P 36e2a5b0519c4f7a892877635a453ce4 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "36e2a5b0519c4f7a892877635a453ce4" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 44283 } }
I20260620 11:46:26.319240 27643 leader_election.cc:302] T 00000000000000000000000000000000 P 36e2a5b0519c4f7a892877635a453ce4 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 36e2a5b0519c4f7a892877635a453ce4; no voters:
I20260620 11:46:26.319386 7083 external_mini_cluster.cc:1388] Started /tmp/dist-test-task1JRg7N/build/release/bin/kudu as pid 27575
I20260620 11:46:26.319420 27643 leader_election.cc:288] T 00000000000000000000000000000000 P 36e2a5b0519c4f7a892877635a453ce4 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260620 11:46:26.319487 27646 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 36e2a5b0519c4f7a892877635a453ce4 [term 1 FOLLOWER]: Leader election won for term 1
I20260620 11:46:26.319499 7083 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task1JRg7N/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1781955746769051-7083-0/minicluster-data/master-0/wal/instance
I20260620 11:46:26.319671 27643 sys_catalog.cc:564] T 00000000000000000000000000000000 P 36e2a5b0519c4f7a892877635a453ce4 [sys.catalog]: configured and running, proceeding with master startup.
I20260620 11:46:26.319712 7083 external_mini_cluster.cc:1428] Setting key 18138f2e6731c59410a41e3aee0e1e4d
I20260620 11:46:26.319743 27646 raft_consensus.cc:686] T 00000000000000000000000000000000 P 36e2a5b0519c4f7a892877635a453ce4 [term 1 LEADER]: Becoming Leader. State: Replica: 36e2a5b0519c4f7a892877635a453ce4, State: Running, Role: LEADER
I20260620 11:46:26.319849 27646 consensus_queue.cc:237] T 00000000000000000000000000000000 P 36e2a5b0519c4f7a892877635a453ce4 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "36e2a5b0519c4f7a892877635a453ce4" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 44283 } }
I20260620 11:46:26.320263 27647 sys_catalog.cc:455] T 00000000000000000000000000000000 P 36e2a5b0519c4f7a892877635a453ce4 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "36e2a5b0519c4f7a892877635a453ce4" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "36e2a5b0519c4f7a892877635a453ce4" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 44283 } } }
I20260620 11:46:26.320374 27648 sys_catalog.cc:455] T 00000000000000000000000000000000 P 36e2a5b0519c4f7a892877635a453ce4 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 36e2a5b0519c4f7a892877635a453ce4. Latest consensus state: current_term: 1 leader_uuid: "36e2a5b0519c4f7a892877635a453ce4" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "36e2a5b0519c4f7a892877635a453ce4" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 44283 } } }
I20260620 11:46:26.320535 27648 sys_catalog.cc:458] T 00000000000000000000000000000000 P 36e2a5b0519c4f7a892877635a453ce4 [sys.catalog]: This master's current role is: LEADER
I20260620 11:46:26.320749 27647 sys_catalog.cc:458] T 00000000000000000000000000000000 P 36e2a5b0519c4f7a892877635a453ce4 [sys.catalog]: This master's current role is: LEADER
I20260620 11:46:26.321117 27656 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260620 11:46:26.321739 27656 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260620 11:46:26.323128 27656 catalog_manager.cc:1347] Generated new cluster ID: e0623518173a4cea9d773d0bdc4d398a
I20260620 11:46:26.323177 27656 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
Jun 20 11:46:26 dist-test-slave-cjc6 krb5kdc[27562](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781955986, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.0.1@KRBTEST.COM
I20260620 11:46:26.332017 27651 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 11:46:26.320230 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43631 (local address 127.0.0.1:44283)
0620 11:46:26.320553 (+ 323us) server_negotiation.cc:206] Beginning negotiation
0620 11:46:26.320564 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0620 11:46:26.320602 (+ 38us) server_negotiation.cc:407] Connection header received
0620 11:46:26.320731 (+ 129us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 11:46:26.320738 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 11:46:26.320816 (+ 78us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 11:46:26.321000 (+ 184us) server_negotiation.cc:226] Negotiated authn=SASL
0620 11:46:26.321811 (+ 811us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 11:46:26.323189 (+ 1378us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 11:46:26.324216 (+ 1027us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 11:46:26.324832 (+ 616us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 11:46:26.324945 (+ 113us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 11:46:26.327723 (+ 2778us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 11:46:26.327727 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 11:46:26.327727 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 11:46:26.327739 (+ 12us) server_negotiation.cc:922] calling sasl_server_start()
0620 11:46:26.330302 (+ 2563us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 11:46:26.330694 (+ 392us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 11:46:26.330697 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 11:46:26.330697 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 11:46:26.330807 (+ 110us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 11:46:26.331013 (+ 206us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 11:46:26.331014 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 11:46:26.331014 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 11:46:26.331309 (+ 295us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 11:46:26.331453 (+ 144us) server_negotiation.cc:1035] Waiting for connection context
0620 11:46:26.331561 (+ 108us) server_negotiation.cc:299] Negotiation successful
0620 11:46:26.331784 (+ 223us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":266,"thread_start_us":70,"threads_started":1}
I20260620 11:46:26.333226 7083 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260620 11:46:26.334720 27656 catalog_manager.cc:1370] Generated new certificate authority record
I20260620 11:46:26.335201 27656 catalog_manager.cc:1504] Loading token signing keys...
I20260620 11:46:26.340148 27656 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 36e2a5b0519c4f7a892877635a453ce4: Generated new TSK 0
I20260620 11:46:26.340327 27656 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260620 11:46:26.346572 27651 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 11:46:26.334760 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52457 (local address 127.0.0.1:44283)
0620 11:46:26.334793 (+ 33us) server_negotiation.cc:206] Beginning negotiation
0620 11:46:26.334795 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 11:46:26.339991 (+ 5196us) server_negotiation.cc:407] Connection header received
0620 11:46:26.340047 (+ 56us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 11:46:26.340049 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 11:46:26.340059 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 11:46:26.340112 (+ 53us) server_negotiation.cc:226] Negotiated authn=SASL
0620 11:46:26.340776 (+ 664us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 11:46:26.341252 (+ 476us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 11:46:26.342518 (+ 1266us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 11:46:26.342726 (+ 208us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 11:46:26.342780 (+ 54us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 11:46:26.344124 (+ 1344us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 11:46:26.344125 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 11:46:26.344126 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 11:46:26.344131 (+ 5us) server_negotiation.cc:922] calling sasl_server_start()
0620 11:46:26.345439 (+ 1308us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 11:46:26.345894 (+ 455us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 11:46:26.345895 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 11:46:26.345895 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 11:46:26.345952 (+ 57us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 11:46:26.346158 (+ 206us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 11:46:26.346158 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 11:46:26.346158 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 11:46:26.346254 (+ 96us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 11:46:26.346330 (+ 76us) server_negotiation.cc:1035] Waiting for connection context
0620 11:46:26.346448 (+ 118us) server_negotiation.cc:299] Negotiation successful
0620 11:46:26.346510 (+ 62us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":16}
I20260620 11:46:26.348421 7083 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task1JRg7N/build/release/bin/kudu with pid 27575
2026-06-20T11:46:26Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/3 (1311 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/4
Loading random data
Initializing database '/tmp/dist-test-task1JRg7N/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1781955746769051-7083-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 20 11:46:26 dist-test-slave-cjc6 krb5kdc[27674](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 20 11:46:26 dist-test-slave-cjc6 krb5kdc[27674](info): set up 2 sockets
Jun 20 11:46:26 dist-test-slave-cjc6 krb5kdc[27674](info): commencing operation
krb5kdc: starting...
W20260620 11:46:27.385834 7083 mini_kdc.cc:121] Time spent starting KDC: real 1.020s user 0.000s sys 0.006s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task1JRg7N/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1781955746769051-7083-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task1JRg7N/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1781955746769051-7083-0/krb5kdc/test-user.keytab.
Jun 20 11:46:27 dist-test-slave-cjc6 krb5kdc[27674](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781955987, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-20T11:46:27Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-20T11:46:27Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task1JRg7N/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1781955746769051-7083-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task1JRg7N/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1781955746769051-7083-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260620 11:46:27.502357 7083 external_mini_cluster.cc:1326] Running /tmp/dist-test-task1JRg7N/build/release/bin/kudu
/tmp/dist-test-task1JRg7N/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task1JRg7N/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1781955746769051-7083-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task1JRg7N/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1781955746769051-7083-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task1JRg7N/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1781955746769051-7083-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task1JRg7N/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1781955746769051-7083-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:39311
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-task1JRg7N/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1781955746769051-7083-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:45249
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:39311
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=required
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-task1JRg7N/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1781955746769051-7083-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task1JRg7N/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1781955746769051-7083-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task1JRg7N/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1781955746769051-7083-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 11:46:27.592350 27687 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 11:46:27.592573 27687 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 11:46:27.592599 27687 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 11:46:27.593875 27687 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 11:46:27.593918 27687 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 11:46:27.593935 27687 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 11:46:27.593948 27687 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 11:46:27.593961 27687 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 11:46:27.595518 27687 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:45249
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task1JRg7N/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1781955746769051-7083-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task1JRg7N/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1781955746769051-7083-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:39311
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task1JRg7N/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1781955746769051-7083-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:39311
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task1JRg7N/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1781955746769051-7083-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.27687
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task1JRg7N/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1781955746769051-7083-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 20 Jun 2026 10:32:37 UTC on ccycloud.quasar-bgyofv.root.comops.site
build id 4457173
I20260620 11:46:27.595736 27687 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 11:46:27.595925 27687 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 11:46:27.601253 27692 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 11:46:27.601215 27695 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 11:46:27.601296 27693 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 11:46:27.601541 27687 server_base.cc:1034] running on GCE node
I20260620 11:46:27.601807 27687 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 11:46:27.602103 27687 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 11:46:27.603256 27687 hybrid_clock.cc:648] HybridClock initialized: now 1781955987603233 us; error 36 us; skew 500 ppm
Jun 20 11:46:27 dist-test-slave-cjc6 krb5kdc[27674](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781955987, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 11:46:27.605690 27687 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260620 11:46:27.606235 27687 webserver.cc:460] Webserver started at http://127.0.0.1:46293/ using document root <none> and password file <none>
I20260620 11:46:27.606463 27687 fs_manager.cc:359] Metadata directory not provided
I20260620 11:46:27.606530 27687 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 11:46:27.606691 27687 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 11:46:27.607745 27687 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task1JRg7N/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1781955746769051-7083-0/minicluster-data/master-0/data/instance:
uuid: "7e0be3451b9b4614bf81d4be65d18677"
format_stamp: "Formatted at 2026-06-20 11:46:27 on dist-test-slave-cjc6"
server_key: "0127cb95ee405877c2e83b224a95d16b"
server_key_iv: "06ce3180ba5df7658471a2f0046a0b1a"
server_key_version: "encryptionkey@0"
I20260620 11:46:27.608122 27687 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task1JRg7N/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1781955746769051-7083-0/minicluster-data/master-0/wal/instance:
uuid: "7e0be3451b9b4614bf81d4be65d18677"
format_stamp: "Formatted at 2026-06-20 11:46:27 on dist-test-slave-cjc6"
server_key: "0127cb95ee405877c2e83b224a95d16b"
server_key_iv: "06ce3180ba5df7658471a2f0046a0b1a"
server_key_version: "encryptionkey@0"
I20260620 11:46:27.609433 27687 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.001s
I20260620 11:46:27.610256 27702 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 11:46:27.610446 27687 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.000s
I20260620 11:46:27.610543 27687 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task1JRg7N/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1781955746769051-7083-0/minicluster-data/master-0/data,/tmp/dist-test-task1JRg7N/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1781955746769051-7083-0/minicluster-data/master-0/wal
uuid: "7e0be3451b9b4614bf81d4be65d18677"
format_stamp: "Formatted at 2026-06-20 11:46:27 on dist-test-slave-cjc6"
server_key: "0127cb95ee405877c2e83b224a95d16b"
server_key_iv: "06ce3180ba5df7658471a2f0046a0b1a"
server_key_version: "encryptionkey@0"
I20260620 11:46:27.610630 27687 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task1JRg7N/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1781955746769051-7083-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task1JRg7N/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1781955746769051-7083-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task1JRg7N/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1781955746769051-7083-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 11:46:27.621028 27687 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 11:46:27.622865 27687 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 11:46:27.623023 27687 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 11:46:27.627132 27687 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:39311
I20260620 11:46:27.627163 27754 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:39311 every 8 connection(s)
I20260620 11:46:27.627490 27687 server_base.cc:1166] Dumped server information to /tmp/dist-test-task1JRg7N/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1781955746769051-7083-0/minicluster-data/master-0/data/info.pb
I20260620 11:46:27.628384 27755 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 11:46:27.630733 27755 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 7e0be3451b9b4614bf81d4be65d18677: Bootstrap starting.
I20260620 11:46:27.631314 27755 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 7e0be3451b9b4614bf81d4be65d18677: Neither blocks nor log segments found. Creating new log.
I20260620 11:46:27.631594 27755 log.cc:824] T 00000000000000000000000000000000 P 7e0be3451b9b4614bf81d4be65d18677: Log is configured to *not* fsync() on all Append() calls
I20260620 11:46:27.632359 27755 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 7e0be3451b9b4614bf81d4be65d18677: No bootstrap required, opened a new log
I20260620 11:46:27.634202 27755 raft_consensus.cc:348] T 00000000000000000000000000000000 P 7e0be3451b9b4614bf81d4be65d18677 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7e0be3451b9b4614bf81d4be65d18677" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 39311 } }
I20260620 11:46:27.634348 27755 raft_consensus.cc:374] T 00000000000000000000000000000000 P 7e0be3451b9b4614bf81d4be65d18677 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 11:46:27.634400 27755 raft_consensus.cc:729] T 00000000000000000000000000000000 P 7e0be3451b9b4614bf81d4be65d18677 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7e0be3451b9b4614bf81d4be65d18677, State: Initialized, Role: FOLLOWER
I20260620 11:46:27.634564 27755 consensus_queue.cc:260] T 00000000000000000000000000000000 P 7e0be3451b9b4614bf81d4be65d18677 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7e0be3451b9b4614bf81d4be65d18677" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 39311 } }
I20260620 11:46:27.634631 27755 raft_consensus.cc:388] T 00000000000000000000000000000000 P 7e0be3451b9b4614bf81d4be65d18677 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260620 11:46:27.634676 27755 raft_consensus.cc:482] T 00000000000000000000000000000000 P 7e0be3451b9b4614bf81d4be65d18677 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260620 11:46:27.634730 27755 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 7e0be3451b9b4614bf81d4be65d18677 [term 0 FOLLOWER]: Advancing to term 1
I20260620 11:46:27.635279 27755 raft_consensus.cc:504] T 00000000000000000000000000000000 P 7e0be3451b9b4614bf81d4be65d18677 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7e0be3451b9b4614bf81d4be65d18677" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 39311 } }
I20260620 11:46:27.635406 27755 leader_election.cc:302] T 00000000000000000000000000000000 P 7e0be3451b9b4614bf81d4be65d18677 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 7e0be3451b9b4614bf81d4be65d18677; no voters:
I20260620 11:46:27.635600 27755 leader_election.cc:288] T 00000000000000000000000000000000 P 7e0be3451b9b4614bf81d4be65d18677 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260620 11:46:27.635630 27758 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 7e0be3451b9b4614bf81d4be65d18677 [term 1 FOLLOWER]: Leader election won for term 1
I20260620 11:46:27.635800 27758 raft_consensus.cc:686] T 00000000000000000000000000000000 P 7e0be3451b9b4614bf81d4be65d18677 [term 1 LEADER]: Becoming Leader. State: Replica: 7e0be3451b9b4614bf81d4be65d18677, State: Running, Role: LEADER
I20260620 11:46:27.635919 27755 sys_catalog.cc:564] T 00000000000000000000000000000000 P 7e0be3451b9b4614bf81d4be65d18677 [sys.catalog]: configured and running, proceeding with master startup.
I20260620 11:46:27.635938 27758 consensus_queue.cc:237] T 00000000000000000000000000000000 P 7e0be3451b9b4614bf81d4be65d18677 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7e0be3451b9b4614bf81d4be65d18677" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 39311 } }
I20260620 11:46:27.636408 27760 sys_catalog.cc:455] T 00000000000000000000000000000000 P 7e0be3451b9b4614bf81d4be65d18677 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 7e0be3451b9b4614bf81d4be65d18677. Latest consensus state: current_term: 1 leader_uuid: "7e0be3451b9b4614bf81d4be65d18677" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7e0be3451b9b4614bf81d4be65d18677" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 39311 } } }
I20260620 11:46:27.636534 27759 sys_catalog.cc:455] T 00000000000000000000000000000000 P 7e0be3451b9b4614bf81d4be65d18677 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "7e0be3451b9b4614bf81d4be65d18677" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7e0be3451b9b4614bf81d4be65d18677" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 39311 } } }
I20260620 11:46:27.636555 27760 sys_catalog.cc:458] T 00000000000000000000000000000000 P 7e0be3451b9b4614bf81d4be65d18677 [sys.catalog]: This master's current role is: LEADER
I20260620 11:46:27.636837 27767 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260620 11:46:27.637033 27759 sys_catalog.cc:458] T 00000000000000000000000000000000 P 7e0be3451b9b4614bf81d4be65d18677 [sys.catalog]: This master's current role is: LEADER
I20260620 11:46:27.637200 7083 external_mini_cluster.cc:1388] Started /tmp/dist-test-task1JRg7N/build/release/bin/kudu as pid 27687
I20260620 11:46:27.637355 7083 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task1JRg7N/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1781955746769051-7083-0/minicluster-data/master-0/wal/instance
I20260620 11:46:27.637346 27767 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260620 11:46:27.637596 7083 external_mini_cluster.cc:1428] Setting key 2b0de1bfc46a725de8c2110860bffb41
I20260620 11:46:27.638840 27767 catalog_manager.cc:1347] Generated new cluster ID: 269166749c0948e3b79c26d23a6447ed
I20260620 11:46:27.638893 27767 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260620 11:46:27.643206 27777 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 11:46:27.638100 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36481 (local address 127.0.0.1:39311)
0620 11:46:27.638266 (+ 166us) server_negotiation.cc:206] Beginning negotiation
0620 11:46:27.638276 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0620 11:46:27.638527 (+ 251us) server_negotiation.cc:407] Connection header received
0620 11:46:27.638645 (+ 118us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 11:46:27.638650 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 11:46:27.638710 (+ 60us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 11:46:27.638880 (+ 170us) server_negotiation.cc:226] Negotiated authn=SASL
0620 11:46:27.639236 (+ 356us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 11:46:27.640308 (+ 1072us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 11:46:27.641225 (+ 917us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 11:46:27.641667 (+ 442us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 11:46:27.641826 (+ 159us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 11:46:27.642528 (+ 702us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 11:46:27.642532 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 11:46:27.642532 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 11:46:27.642718 (+ 186us) server_negotiation.cc:922] calling sasl_server_start()
0620 11:46:27.642748 (+ 30us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 11:46:27.642758 (+ 10us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 11:46:27.642833 (+ 75us) server_negotiation.cc:1035] Waiting for connection context
0620 11:46:27.642928 (+ 95us) server_negotiation.cc:299] Negotiation successful
0620 11:46:27.642991 (+ 63us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":109,"thread_start_us":79,"threads_started":1}
I20260620 11:46:27.644023 7083 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260620 11:46:27.647981 27777 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 11:46:27.645306 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38227 (local address 127.0.0.1:39311)
0620 11:46:27.645341 (+ 35us) server_negotiation.cc:206] Beginning negotiation
0620 11:46:27.645343 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 11:46:27.645402 (+ 59us) server_negotiation.cc:407] Connection header received
0620 11:46:27.645470 (+ 68us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 11:46:27.645471 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 11:46:27.645480 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 11:46:27.645527 (+ 47us) server_negotiation.cc:226] Negotiated authn=SASL
0620 11:46:27.645623 (+ 96us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 11:46:27.646136 (+ 513us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 11:46:27.646821 (+ 685us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 11:46:27.647044 (+ 223us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 11:46:27.647114 (+ 70us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 11:46:27.647742 (+ 628us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 11:46:27.647743 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 11:46:27.647743 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 11:46:27.647748 (+ 5us) server_negotiation.cc:922] calling sasl_server_start()
0620 11:46:27.647760 (+ 12us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 11:46:27.647765 (+ 5us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 11:46:27.647808 (+ 43us) server_negotiation.cc:1035] Waiting for connection context
0620 11:46:27.647868 (+ 60us) server_negotiation.cc:299] Negotiation successful
0620 11:46:27.647888 (+ 20us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":18}
I20260620 11:46:27.672760 27767 catalog_manager.cc:1370] Generated new certificate authority record
I20260620 11:46:27.673316 27767 catalog_manager.cc:1504] Loading token signing keys...
I20260620 11:46:27.677902 27767 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 7e0be3451b9b4614bf81d4be65d18677: Generated new TSK 0
I20260620 11:46:27.678107 27767 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260620 11:46:27.683054 7083 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task1JRg7N/build/release/bin/kudu with pid 27687
2026-06-20T11:46:27Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/4 (1334 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/5
Loading random data
Initializing database '/tmp/dist-test-task1JRg7N/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1781955746769051-7083-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 20 11:46:27 dist-test-slave-cjc6 krb5kdc[27786](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 20 11:46:27 dist-test-slave-cjc6 krb5kdc[27786](info): set up 2 sockets
Jun 20 11:46:27 dist-test-slave-cjc6 krb5kdc[27786](info): commencing operation
krb5kdc: starting...
W20260620 11:46:28.729908 7083 mini_kdc.cc:121] Time spent starting KDC: real 1.029s user 0.005s sys 0.000s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task1JRg7N/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1781955746769051-7083-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task1JRg7N/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1781955746769051-7083-0/krb5kdc/test-user.keytab.
Jun 20 11:46:28 dist-test-slave-cjc6 krb5kdc[27786](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781955988, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-20T11:46:28Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-20T11:46:28Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task1JRg7N/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1781955746769051-7083-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task1JRg7N/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1781955746769051-7083-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260620 11:46:28.848131 7083 external_mini_cluster.cc:1326] Running /tmp/dist-test-task1JRg7N/build/release/bin/kudu
/tmp/dist-test-task1JRg7N/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task1JRg7N/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1781955746769051-7083-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task1JRg7N/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1781955746769051-7083-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task1JRg7N/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1781955746769051-7083-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task1JRg7N/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1781955746769051-7083-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:38939
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-task1JRg7N/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1781955746769051-7083-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:36813
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:38939
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=disabled
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-task1JRg7N/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1781955746769051-7083-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task1JRg7N/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1781955746769051-7083-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task1JRg7N/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1781955746769051-7083-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 11:46:28.928629 27799 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 11:46:28.928902 27799 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 11:46:28.928929 27799 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 11:46:28.930158 27799 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 11:46:28.930202 27799 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 11:46:28.930217 27799 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 11:46:28.930229 27799 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 11:46:28.930289 27799 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 11:46:28.931733 27799 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:36813
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task1JRg7N/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1781955746769051-7083-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task1JRg7N/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1781955746769051-7083-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:38939
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task1JRg7N/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1781955746769051-7083-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:38939
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task1JRg7N/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1781955746769051-7083-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.27799
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task1JRg7N/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1781955746769051-7083-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 20 Jun 2026 10:32:37 UTC on ccycloud.quasar-bgyofv.root.comops.site
build id 4457173
I20260620 11:46:28.932060 27799 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 11:46:28.932317 27799 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 11:46:28.937179 27805 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 11:46:28.937184 27804 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 11:46:28.937227 27807 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 11:46:28.937441 27799 server_base.cc:1034] running on GCE node
I20260620 11:46:28.937671 27799 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 11:46:28.937992 27799 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 11:46:28.939150 27799 hybrid_clock.cc:648] HybridClock initialized: now 1781955988939140 us; error 34 us; skew 500 ppm
Jun 20 11:46:28 dist-test-slave-cjc6 krb5kdc[27786](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781955988, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 11:46:28.941494 27799 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260620 11:46:28.941995 27799 webserver.cc:460] Webserver started at http://127.0.0.1:44785/ using document root <none> and password file <none>
I20260620 11:46:28.942242 27799 fs_manager.cc:359] Metadata directory not provided
I20260620 11:46:28.942310 27799 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 11:46:28.942463 27799 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 11:46:28.943498 27799 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task1JRg7N/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1781955746769051-7083-0/minicluster-data/master-0/data/instance:
uuid: "7d7a318ea8844f02a6031670483d4ede"
format_stamp: "Formatted at 2026-06-20 11:46:28 on dist-test-slave-cjc6"
server_key: "b748d00bbd79ce96221c50a9487108f3"
server_key_iv: "ededda5fb189d567b7af2f7bd87a6c0f"
server_key_version: "encryptionkey@0"
I20260620 11:46:28.943861 27799 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task1JRg7N/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1781955746769051-7083-0/minicluster-data/master-0/wal/instance:
uuid: "7d7a318ea8844f02a6031670483d4ede"
format_stamp: "Formatted at 2026-06-20 11:46:28 on dist-test-slave-cjc6"
server_key: "b748d00bbd79ce96221c50a9487108f3"
server_key_iv: "ededda5fb189d567b7af2f7bd87a6c0f"
server_key_version: "encryptionkey@0"
I20260620 11:46:28.945184 27799 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260620 11:46:28.945981 27814 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 11:46:28.946161 27799 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260620 11:46:28.946266 27799 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task1JRg7N/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1781955746769051-7083-0/minicluster-data/master-0/data,/tmp/dist-test-task1JRg7N/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1781955746769051-7083-0/minicluster-data/master-0/wal
uuid: "7d7a318ea8844f02a6031670483d4ede"
format_stamp: "Formatted at 2026-06-20 11:46:28 on dist-test-slave-cjc6"
server_key: "b748d00bbd79ce96221c50a9487108f3"
server_key_iv: "ededda5fb189d567b7af2f7bd87a6c0f"
server_key_version: "encryptionkey@0"
I20260620 11:46:28.946342 27799 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task1JRg7N/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1781955746769051-7083-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task1JRg7N/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1781955746769051-7083-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task1JRg7N/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1781955746769051-7083-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 11:46:28.949579 27799 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 11:46:28.951489 27799 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 11:46:28.951649 27799 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 11:46:28.955554 27799 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:38939
I20260620 11:46:28.955590 27866 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:38939 every 8 connection(s)
I20260620 11:46:28.955917 27799 server_base.cc:1166] Dumped server information to /tmp/dist-test-task1JRg7N/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1781955746769051-7083-0/minicluster-data/master-0/data/info.pb
I20260620 11:46:28.956879 27867 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 11:46:28.959125 27867 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 7d7a318ea8844f02a6031670483d4ede: Bootstrap starting.
I20260620 11:46:28.959697 27867 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 7d7a318ea8844f02a6031670483d4ede: Neither blocks nor log segments found. Creating new log.
I20260620 11:46:28.959991 27867 log.cc:824] T 00000000000000000000000000000000 P 7d7a318ea8844f02a6031670483d4ede: Log is configured to *not* fsync() on all Append() calls
I20260620 11:46:28.960656 27867 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 7d7a318ea8844f02a6031670483d4ede: No bootstrap required, opened a new log
I20260620 11:46:28.962272 27867 raft_consensus.cc:348] T 00000000000000000000000000000000 P 7d7a318ea8844f02a6031670483d4ede [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7d7a318ea8844f02a6031670483d4ede" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 38939 } }
I20260620 11:46:28.962421 27867 raft_consensus.cc:374] T 00000000000000000000000000000000 P 7d7a318ea8844f02a6031670483d4ede [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 11:46:28.962445 27867 raft_consensus.cc:729] T 00000000000000000000000000000000 P 7d7a318ea8844f02a6031670483d4ede [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7d7a318ea8844f02a6031670483d4ede, State: Initialized, Role: FOLLOWER
I20260620 11:46:28.962620 27867 consensus_queue.cc:260] T 00000000000000000000000000000000 P 7d7a318ea8844f02a6031670483d4ede [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7d7a318ea8844f02a6031670483d4ede" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 38939 } }
I20260620 11:46:28.962726 27867 raft_consensus.cc:388] T 00000000000000000000000000000000 P 7d7a318ea8844f02a6031670483d4ede [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260620 11:46:28.962771 27867 raft_consensus.cc:482] T 00000000000000000000000000000000 P 7d7a318ea8844f02a6031670483d4ede [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260620 11:46:28.962791 7083 external_mini_cluster.cc:1388] Started /tmp/dist-test-task1JRg7N/build/release/bin/kudu as pid 27799
I20260620 11:46:28.962862 27867 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 7d7a318ea8844f02a6031670483d4ede [term 0 FOLLOWER]: Advancing to term 1
I20260620 11:46:28.962944 7083 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task1JRg7N/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1781955746769051-7083-0/minicluster-data/master-0/wal/instance
I20260620 11:46:28.963166 7083 external_mini_cluster.cc:1428] Setting key 9d62fa219753e4bc08367a83625b22d9
I20260620 11:46:28.963603 27867 raft_consensus.cc:504] T 00000000000000000000000000000000 P 7d7a318ea8844f02a6031670483d4ede [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7d7a318ea8844f02a6031670483d4ede" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 38939 } }
I20260620 11:46:28.963783 27867 leader_election.cc:302] T 00000000000000000000000000000000 P 7d7a318ea8844f02a6031670483d4ede [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 7d7a318ea8844f02a6031670483d4ede; no voters:
I20260620 11:46:28.963953 27867 leader_election.cc:288] T 00000000000000000000000000000000 P 7d7a318ea8844f02a6031670483d4ede [CANDIDATE]: Term 1 election: Requested vote from peers
I20260620 11:46:28.963989 27872 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 7d7a318ea8844f02a6031670483d4ede [term 1 FOLLOWER]: Leader election won for term 1
I20260620 11:46:28.964145 27872 raft_consensus.cc:686] T 00000000000000000000000000000000 P 7d7a318ea8844f02a6031670483d4ede [term 1 LEADER]: Becoming Leader. State: Replica: 7d7a318ea8844f02a6031670483d4ede, State: Running, Role: LEADER
I20260620 11:46:28.964268 27872 consensus_queue.cc:237] T 00000000000000000000000000000000 P 7d7a318ea8844f02a6031670483d4ede [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7d7a318ea8844f02a6031670483d4ede" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 38939 } }
I20260620 11:46:28.964278 27867 sys_catalog.cc:564] T 00000000000000000000000000000000 P 7d7a318ea8844f02a6031670483d4ede [sys.catalog]: configured and running, proceeding with master startup.
I20260620 11:46:28.964633 27872 sys_catalog.cc:455] T 00000000000000000000000000000000 P 7d7a318ea8844f02a6031670483d4ede [sys.catalog]: SysCatalogTable state changed. Reason: New leader 7d7a318ea8844f02a6031670483d4ede. Latest consensus state: current_term: 1 leader_uuid: "7d7a318ea8844f02a6031670483d4ede" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7d7a318ea8844f02a6031670483d4ede" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 38939 } } }
I20260620 11:46:28.964709 27872 sys_catalog.cc:458] T 00000000000000000000000000000000 P 7d7a318ea8844f02a6031670483d4ede [sys.catalog]: This master's current role is: LEADER
I20260620 11:46:28.964867 27873 sys_catalog.cc:455] T 00000000000000000000000000000000 P 7d7a318ea8844f02a6031670483d4ede [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "7d7a318ea8844f02a6031670483d4ede" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7d7a318ea8844f02a6031670483d4ede" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 38939 } } }
I20260620 11:46:28.965000 27873 sys_catalog.cc:458] T 00000000000000000000000000000000 P 7d7a318ea8844f02a6031670483d4ede [sys.catalog]: This master's current role is: LEADER
I20260620 11:46:28.965818 27871 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 11:46:28.963650 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45755 (local address 127.0.0.1:38939)
0620 11:46:28.963827 (+ 177us) server_negotiation.cc:206] Beginning negotiation
0620 11:46:28.963836 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0620 11:46:28.963884 (+ 48us) server_negotiation.cc:407] Connection header received
0620 11:46:28.964022 (+ 138us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 11:46:28.964030 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 11:46:28.964065 (+ 35us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 11:46:28.964212 (+ 147us) server_negotiation.cc:226] Negotiated authn=SASL
0620 11:46:28.965235 (+ 1023us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 11:46:28.965239 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 11:46:28.965240 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 11:46:28.965244 (+ 4us) server_negotiation.cc:922] calling sasl_server_start()
0620 11:46:28.965292 (+ 48us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 11:46:28.965317 (+ 25us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 11:46:28.965414 (+ 97us) server_negotiation.cc:1035] Waiting for connection context
0620 11:46:28.965548 (+ 134us) server_negotiation.cc:299] Negotiation successful
0620 11:46:28.965605 (+ 57us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":123,"thread_start_us":91,"threads_started":1}
W20260620 11:46:28.966624 27888 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 7d7a318ea8844f02a6031670483d4ede: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260620 11:46:28.966713 27888 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260620 11:46:28.966774 7083 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260620 11:46:28.966786 27876 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260620 11:46:28.966922 27876 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260620 11:46:28.968451 27876 catalog_manager.cc:1347] Generated new cluster ID: 1aa61e52411b47c3b0bf608e667b31dd
I20260620 11:46:28.968508 27876 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260620 11:46:28.969240 27871 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 11:46:28.968277 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44285 (local address 127.0.0.1:38939)
0620 11:46:28.968313 (+ 36us) server_negotiation.cc:206] Beginning negotiation
0620 11:46:28.968315 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 11:46:28.968425 (+ 110us) server_negotiation.cc:407] Connection header received
0620 11:46:28.968484 (+ 59us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 11:46:28.968484 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 11:46:28.968489 (+ 5us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 11:46:28.968550 (+ 61us) server_negotiation.cc:226] Negotiated authn=SASL
0620 11:46:28.969078 (+ 528us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 11:46:28.969079 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 11:46:28.969079 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 11:46:28.969083 (+ 4us) server_negotiation.cc:922] calling sasl_server_start()
0620 11:46:28.969094 (+ 11us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 11:46:28.969098 (+ 4us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 11:46:28.969142 (+ 44us) server_negotiation.cc:1035] Waiting for connection context
0620 11:46:28.969190 (+ 48us) server_negotiation.cc:299] Negotiation successful
0620 11:46:28.969200 (+ 10us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":21}
I20260620 11:46:29.002154 27876 catalog_manager.cc:1370] Generated new certificate authority record
I20260620 11:46:29.002369 27876 catalog_manager.cc:1504] Loading token signing keys...
I20260620 11:46:29.006409 27876 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 7d7a318ea8844f02a6031670483d4ede: Generated new TSK 0
I20260620 11:46:29.006605 27876 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260620 11:46:29.011530 7083 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task1JRg7N/build/release/bin/kudu with pid 27799
2026-06-20T11:46:29Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/5 (1328 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/6
Loading random data
Initializing database '/tmp/dist-test-task1JRg7N/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1781955746769051-7083-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 20 11:46:29 dist-test-slave-cjc6 krb5kdc[27898](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 20 11:46:29 dist-test-slave-cjc6 krb5kdc[27898](info): set up 2 sockets
Jun 20 11:46:29 dist-test-slave-cjc6 krb5kdc[27898](info): commencing operation
krb5kdc: starting...
W20260620 11:46:30.057940 7083 mini_kdc.cc:121] Time spent starting KDC: real 1.028s user 0.000s sys 0.006s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task1JRg7N/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1781955746769051-7083-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task1JRg7N/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1781955746769051-7083-0/krb5kdc/test-user.keytab.
Jun 20 11:46:30 dist-test-slave-cjc6 krb5kdc[27898](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781955990, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-20T11:46:30Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-20T11:46:30Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task1JRg7N/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1781955746769051-7083-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task1JRg7N/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1781955746769051-7083-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260620 11:46:30.176095 7083 external_mini_cluster.cc:1326] Running /tmp/dist-test-task1JRg7N/build/release/bin/kudu
/tmp/dist-test-task1JRg7N/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task1JRg7N/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1781955746769051-7083-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task1JRg7N/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1781955746769051-7083-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task1JRg7N/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1781955746769051-7083-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task1JRg7N/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1781955746769051-7083-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:45959
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-task1JRg7N/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1781955746769051-7083-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:33285
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:45959
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=required
--rpc_encryption=required
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-task1JRg7N/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1781955746769051-7083-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task1JRg7N/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1781955746769051-7083-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task1JRg7N/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1781955746769051-7083-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 11:46:30.257018 27911 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 11:46:30.257313 27911 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 11:46:30.257351 27911 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 11:46:30.258550 27911 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 11:46:30.258630 27911 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 11:46:30.258653 27911 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 11:46:30.258677 27911 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 11:46:30.258704 27911 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 11:46:30.260169 27911 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:33285
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task1JRg7N/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1781955746769051-7083-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task1JRg7N/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1781955746769051-7083-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:45959
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task1JRg7N/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1781955746769051-7083-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:45959
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task1JRg7N/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1781955746769051-7083-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.27911
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task1JRg7N/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1781955746769051-7083-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 20 Jun 2026 10:32:37 UTC on ccycloud.quasar-bgyofv.root.comops.site
build id 4457173
I20260620 11:46:30.260450 27911 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 11:46:30.260684 27911 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 11:46:30.265656 27916 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 11:46:30.265672 27917 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 11:46:30.265671 27919 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 11:46:30.265897 27911 server_base.cc:1034] running on GCE node
I20260620 11:46:30.266213 27911 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 11:46:30.266673 27911 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 11:46:30.267894 27911 hybrid_clock.cc:648] HybridClock initialized: now 1781955990267870 us; error 43 us; skew 500 ppm
Jun 20 11:46:30 dist-test-slave-cjc6 krb5kdc[27898](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781955990, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 11:46:30.270449 27911 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260620 11:46:30.270947 27911 webserver.cc:460] Webserver started at http://127.0.0.1:33723/ using document root <none> and password file <none>
I20260620 11:46:30.271196 27911 fs_manager.cc:359] Metadata directory not provided
I20260620 11:46:30.271265 27911 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 11:46:30.271420 27911 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 11:46:30.272502 27911 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task1JRg7N/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1781955746769051-7083-0/minicluster-data/master-0/data/instance:
uuid: "c8aea8a582e846d8947faf3970c8faa0"
format_stamp: "Formatted at 2026-06-20 11:46:30 on dist-test-slave-cjc6"
server_key: "ccfbd95d16a463d5e1c42fbc5889b276"
server_key_iv: "ec17d9956624602272f8e1b3519781ba"
server_key_version: "encryptionkey@0"
I20260620 11:46:30.272859 27911 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task1JRg7N/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1781955746769051-7083-0/minicluster-data/master-0/wal/instance:
uuid: "c8aea8a582e846d8947faf3970c8faa0"
format_stamp: "Formatted at 2026-06-20 11:46:30 on dist-test-slave-cjc6"
server_key: "ccfbd95d16a463d5e1c42fbc5889b276"
server_key_iv: "ec17d9956624602272f8e1b3519781ba"
server_key_version: "encryptionkey@0"
I20260620 11:46:30.274291 27911 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.001s
I20260620 11:46:30.275111 27926 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 11:46:30.275339 27911 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260620 11:46:30.275461 27911 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task1JRg7N/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1781955746769051-7083-0/minicluster-data/master-0/data,/tmp/dist-test-task1JRg7N/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1781955746769051-7083-0/minicluster-data/master-0/wal
uuid: "c8aea8a582e846d8947faf3970c8faa0"
format_stamp: "Formatted at 2026-06-20 11:46:30 on dist-test-slave-cjc6"
server_key: "ccfbd95d16a463d5e1c42fbc5889b276"
server_key_iv: "ec17d9956624602272f8e1b3519781ba"
server_key_version: "encryptionkey@0"
I20260620 11:46:30.275538 27911 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task1JRg7N/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1781955746769051-7083-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task1JRg7N/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1781955746769051-7083-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task1JRg7N/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1781955746769051-7083-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 11:46:30.292600 27911 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 11:46:30.294735 27911 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 11:46:30.294906 27911 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 11:46:30.299858 27911 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:45959
I20260620 11:46:30.299908 27978 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:45959 every 8 connection(s)
I20260620 11:46:30.300420 27911 server_base.cc:1166] Dumped server information to /tmp/dist-test-task1JRg7N/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1781955746769051-7083-0/minicluster-data/master-0/data/info.pb
I20260620 11:46:30.301420 7083 external_mini_cluster.cc:1388] Started /tmp/dist-test-task1JRg7N/build/release/bin/kudu as pid 27911
I20260620 11:46:30.301471 27979 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 11:46:30.301561 7083 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task1JRg7N/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1781955746769051-7083-0/minicluster-data/master-0/wal/instance
I20260620 11:46:30.301791 7083 external_mini_cluster.cc:1428] Setting key e6d1f3773c8e49ffcbee059672a3985c
I20260620 11:46:30.305089 27979 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P c8aea8a582e846d8947faf3970c8faa0: Bootstrap starting.
I20260620 11:46:30.305922 27979 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P c8aea8a582e846d8947faf3970c8faa0: Neither blocks nor log segments found. Creating new log.
I20260620 11:46:30.306257 27979 log.cc:824] T 00000000000000000000000000000000 P c8aea8a582e846d8947faf3970c8faa0: Log is configured to *not* fsync() on all Append() calls
I20260620 11:46:30.307308 27979 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P c8aea8a582e846d8947faf3970c8faa0: No bootstrap required, opened a new log
I20260620 11:46:30.309823 27979 raft_consensus.cc:348] T 00000000000000000000000000000000 P c8aea8a582e846d8947faf3970c8faa0 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c8aea8a582e846d8947faf3970c8faa0" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 45959 } }
I20260620 11:46:30.310042 27979 raft_consensus.cc:374] T 00000000000000000000000000000000 P c8aea8a582e846d8947faf3970c8faa0 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 11:46:30.310096 27979 raft_consensus.cc:729] T 00000000000000000000000000000000 P c8aea8a582e846d8947faf3970c8faa0 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c8aea8a582e846d8947faf3970c8faa0, State: Initialized, Role: FOLLOWER
I20260620 11:46:30.310289 27979 consensus_queue.cc:260] T 00000000000000000000000000000000 P c8aea8a582e846d8947faf3970c8faa0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c8aea8a582e846d8947faf3970c8faa0" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 45959 } }
I20260620 11:46:30.310389 27979 raft_consensus.cc:388] T 00000000000000000000000000000000 P c8aea8a582e846d8947faf3970c8faa0 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260620 11:46:30.310427 27979 raft_consensus.cc:482] T 00000000000000000000000000000000 P c8aea8a582e846d8947faf3970c8faa0 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260620 11:46:30.310478 27979 raft_consensus.cc:3037] T 00000000000000000000000000000000 P c8aea8a582e846d8947faf3970c8faa0 [term 0 FOLLOWER]: Advancing to term 1
I20260620 11:46:30.311301 27979 raft_consensus.cc:504] T 00000000000000000000000000000000 P c8aea8a582e846d8947faf3970c8faa0 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c8aea8a582e846d8947faf3970c8faa0" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 45959 } }
I20260620 11:46:30.311420 27979 leader_election.cc:302] T 00000000000000000000000000000000 P c8aea8a582e846d8947faf3970c8faa0 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: c8aea8a582e846d8947faf3970c8faa0; no voters:
I20260620 11:46:30.311589 27979 leader_election.cc:288] T 00000000000000000000000000000000 P c8aea8a582e846d8947faf3970c8faa0 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260620 11:46:30.311683 27984 raft_consensus.cc:2781] T 00000000000000000000000000000000 P c8aea8a582e846d8947faf3970c8faa0 [term 1 FOLLOWER]: Leader election won for term 1
I20260620 11:46:30.311897 27979 sys_catalog.cc:564] T 00000000000000000000000000000000 P c8aea8a582e846d8947faf3970c8faa0 [sys.catalog]: configured and running, proceeding with master startup.
Jun 20 11:46:30 dist-test-slave-cjc6 krb5kdc[27898](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781955990, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.0.1@KRBTEST.COM
I20260620 11:46:30.312141 27984 raft_consensus.cc:686] T 00000000000000000000000000000000 P c8aea8a582e846d8947faf3970c8faa0 [term 1 LEADER]: Becoming Leader. State: Replica: c8aea8a582e846d8947faf3970c8faa0, State: Running, Role: LEADER
I20260620 11:46:30.312278 27984 consensus_queue.cc:237] T 00000000000000000000000000000000 P c8aea8a582e846d8947faf3970c8faa0 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c8aea8a582e846d8947faf3970c8faa0" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 45959 } }
I20260620 11:46:30.312703 27984 sys_catalog.cc:455] T 00000000000000000000000000000000 P c8aea8a582e846d8947faf3970c8faa0 [sys.catalog]: SysCatalogTable state changed. Reason: New leader c8aea8a582e846d8947faf3970c8faa0. Latest consensus state: current_term: 1 leader_uuid: "c8aea8a582e846d8947faf3970c8faa0" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c8aea8a582e846d8947faf3970c8faa0" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 45959 } } }
I20260620 11:46:30.312804 27986 sys_catalog.cc:455] T 00000000000000000000000000000000 P c8aea8a582e846d8947faf3970c8faa0 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "c8aea8a582e846d8947faf3970c8faa0" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c8aea8a582e846d8947faf3970c8faa0" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 45959 } } }
I20260620 11:46:30.312891 27986 sys_catalog.cc:458] T 00000000000000000000000000000000 P c8aea8a582e846d8947faf3970c8faa0 [sys.catalog]: This master's current role is: LEADER
I20260620 11:46:30.313090 27993 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260620 11:46:30.312835 27984 sys_catalog.cc:458] T 00000000000000000000000000000000 P c8aea8a582e846d8947faf3970c8faa0 [sys.catalog]: This master's current role is: LEADER
I20260620 11:46:30.313916 27993 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260620 11:46:30.315593 27993 catalog_manager.cc:1347] Generated new cluster ID: 69685a6297a04c4d9ae428d575254627
I20260620 11:46:30.315644 27993 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260620 11:46:30.317221 27982 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 11:46:30.303002 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.133:40243 (local address 127.0.0.1:45959)
0620 11:46:30.303172 (+ 170us) server_negotiation.cc:206] Beginning negotiation
0620 11:46:30.303183 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0620 11:46:30.303221 (+ 38us) server_negotiation.cc:407] Connection header received
0620 11:46:30.304096 (+ 875us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 11:46:30.304106 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 11:46:30.304179 (+ 73us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 11:46:30.304355 (+ 176us) server_negotiation.cc:226] Negotiated authn=SASL
0620 11:46:30.305259 (+ 904us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 11:46:30.306997 (+ 1738us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 11:46:30.308298 (+ 1301us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 11:46:30.309169 (+ 871us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 11:46:30.309321 (+ 152us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 11:46:30.312611 (+ 3290us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 11:46:30.312618 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 11:46:30.312618 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 11:46:30.312631 (+ 13us) server_negotiation.cc:922] calling sasl_server_start()
0620 11:46:30.315153 (+ 2522us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 11:46:30.315699 (+ 546us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 11:46:30.315702 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 11:46:30.315703 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 11:46:30.315801 (+ 98us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 11:46:30.316035 (+ 234us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 11:46:30.316035 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 11:46:30.316036 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 11:46:30.316319 (+ 283us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 11:46:30.316848 (+ 529us) server_negotiation.cc:1035] Waiting for connection context
0620 11:46:30.316929 (+ 81us) server_negotiation.cc:299] Negotiation successful
0620 11:46:30.317021 (+ 92us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":117,"thread_start_us":73,"threads_started":1}
I20260620 11:46:30.318156 7083 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260620 11:46:30.320540 27993 catalog_manager.cc:1370] Generated new certificate authority record
I20260620 11:46:30.321228 27993 catalog_manager.cc:1504] Loading token signing keys...
I20260620 11:46:30.329249 27982 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 11:46:30.319595 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.133:40107 (local address 127.0.0.1:45959)
0620 11:46:30.319640 (+ 45us) server_negotiation.cc:206] Beginning negotiation
0620 11:46:30.319642 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 11:46:30.319772 (+ 130us) server_negotiation.cc:407] Connection header received
0620 11:46:30.320313 (+ 541us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 11:46:30.320314 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 11:46:30.320324 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 11:46:30.320401 (+ 77us) server_negotiation.cc:226] Negotiated authn=SASL
0620 11:46:30.322879 (+ 2478us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 11:46:30.323638 (+ 759us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 11:46:30.324619 (+ 981us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 11:46:30.324957 (+ 338us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 11:46:30.325032 (+ 75us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 11:46:30.326424 (+ 1392us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 11:46:30.326425 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 11:46:30.326426 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 11:46:30.326439 (+ 13us) server_negotiation.cc:922] calling sasl_server_start()
0620 11:46:30.328041 (+ 1602us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 11:46:30.328455 (+ 414us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 11:46:30.328456 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 11:46:30.328456 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 11:46:30.328515 (+ 59us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 11:46:30.328736 (+ 221us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 11:46:30.328738 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 11:46:30.328738 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 11:46:30.328862 (+ 124us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 11:46:30.328962 (+ 100us) server_negotiation.cc:1035] Waiting for connection context
0620 11:46:30.329091 (+ 129us) server_negotiation.cc:299] Negotiation successful
0620 11:46:30.329154 (+ 63us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":28}
I20260620 11:46:30.331684 27993 catalog_manager.cc:5868] T 00000000000000000000000000000000 P c8aea8a582e846d8947faf3970c8faa0: Generated new TSK 0
I20260620 11:46:30.331892 27993 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260620 11:46:30.334646 7083 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task1JRg7N/build/release/bin/kudu with pid 27911
W20260620 11:46:30.342412 27896 connection.cc:535] client connection to 127.0.0.1:45959 recv error: Network error: failed to read from TLS socket (remote: 127.0.0.1:45959): Connection reset by peer (error 104)
2026-06-20T11:46:30Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/6 (1326 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/7
Loading random data
Initializing database '/tmp/dist-test-task1JRg7N/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1781955746769051-7083-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 20 11:46:30 dist-test-slave-cjc6 krb5kdc[28010](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 20 11:46:30 dist-test-slave-cjc6 krb5kdc[28010](info): set up 2 sockets
Jun 20 11:46:30 dist-test-slave-cjc6 krb5kdc[28010](info): commencing operation
krb5kdc: starting...
W20260620 11:46:31.377791 7083 mini_kdc.cc:121] Time spent starting KDC: real 1.021s user 0.000s sys 0.006s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task1JRg7N/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1781955746769051-7083-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task1JRg7N/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1781955746769051-7083-0/krb5kdc/test-user.keytab.
Jun 20 11:46:31 dist-test-slave-cjc6 krb5kdc[28010](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781955991, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-20T11:46:31Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-20T11:46:31Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task1JRg7N/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1781955746769051-7083-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task1JRg7N/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1781955746769051-7083-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260620 11:46:31.492791 7083 external_mini_cluster.cc:1326] Running /tmp/dist-test-task1JRg7N/build/release/bin/kudu
/tmp/dist-test-task1JRg7N/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task1JRg7N/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1781955746769051-7083-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task1JRg7N/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1781955746769051-7083-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task1JRg7N/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1781955746769051-7083-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task1JRg7N/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1781955746769051-7083-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:33349
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-task1JRg7N/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1781955746769051-7083-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:40769
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:33349
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=required
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-task1JRg7N/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1781955746769051-7083-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task1JRg7N/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1781955746769051-7083-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task1JRg7N/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1781955746769051-7083-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 11:46:31.573747 28023 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 11:46:31.573994 28023 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 11:46:31.574033 28023 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 11:46:31.575176 28023 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 11:46:31.575249 28023 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 11:46:31.575275 28023 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 11:46:31.575309 28023 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 11:46:31.575340 28023 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 11:46:31.576818 28023 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:40769
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task1JRg7N/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1781955746769051-7083-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task1JRg7N/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1781955746769051-7083-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:33349
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task1JRg7N/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1781955746769051-7083-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:33349
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task1JRg7N/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1781955746769051-7083-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.28023
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task1JRg7N/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1781955746769051-7083-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 20 Jun 2026 10:32:37 UTC on ccycloud.quasar-bgyofv.root.comops.site
build id 4457173
I20260620 11:46:31.577090 28023 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 11:46:31.577315 28023 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 11:46:31.582087 28029 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 11:46:31.582132 28031 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 11:46:31.582257 28023 server_base.cc:1034] running on GCE node
W20260620 11:46:31.582412 28028 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 11:46:31.582643 28023 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 11:46:31.582895 28023 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 11:46:31.584036 28023 hybrid_clock.cc:648] HybridClock initialized: now 1781955991584015 us; error 41 us; skew 500 ppm
Jun 20 11:46:31 dist-test-slave-cjc6 krb5kdc[28010](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781955991, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 11:46:31.590607 28023 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260620 11:46:31.591099 28023 webserver.cc:460] Webserver started at http://127.0.0.1:33615/ using document root <none> and password file <none>
I20260620 11:46:31.591346 28023 fs_manager.cc:359] Metadata directory not provided
I20260620 11:46:31.591419 28023 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 11:46:31.591552 28023 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 11:46:31.592741 28023 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task1JRg7N/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1781955746769051-7083-0/minicluster-data/master-0/data/instance:
uuid: "41db0c1d13cf4bf4bed55997733bcbef"
format_stamp: "Formatted at 2026-06-20 11:46:31 on dist-test-slave-cjc6"
server_key: "61cfee711b4e5ba9fbdf92b14ae6879c"
server_key_iv: "f123baccc7c0c82961724f6f69915ff8"
server_key_version: "encryptionkey@0"
I20260620 11:46:31.593114 28023 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task1JRg7N/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1781955746769051-7083-0/minicluster-data/master-0/wal/instance:
uuid: "41db0c1d13cf4bf4bed55997733bcbef"
format_stamp: "Formatted at 2026-06-20 11:46:31 on dist-test-slave-cjc6"
server_key: "61cfee711b4e5ba9fbdf92b14ae6879c"
server_key_iv: "f123baccc7c0c82961724f6f69915ff8"
server_key_version: "encryptionkey@0"
I20260620 11:46:31.594568 28023 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260620 11:46:31.595363 28038 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 11:46:31.595548 28023 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260620 11:46:31.595649 28023 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task1JRg7N/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1781955746769051-7083-0/minicluster-data/master-0/data,/tmp/dist-test-task1JRg7N/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1781955746769051-7083-0/minicluster-data/master-0/wal
uuid: "41db0c1d13cf4bf4bed55997733bcbef"
format_stamp: "Formatted at 2026-06-20 11:46:31 on dist-test-slave-cjc6"
server_key: "61cfee711b4e5ba9fbdf92b14ae6879c"
server_key_iv: "f123baccc7c0c82961724f6f69915ff8"
server_key_version: "encryptionkey@0"
I20260620 11:46:31.595738 28023 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task1JRg7N/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1781955746769051-7083-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task1JRg7N/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1781955746769051-7083-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task1JRg7N/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1781955746769051-7083-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 11:46:31.609668 28023 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 11:46:31.611696 28023 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 11:46:31.611959 28023 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 11:46:31.616173 28023 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:33349
I20260620 11:46:31.616209 28090 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:33349 every 8 connection(s)
I20260620 11:46:31.616555 28023 server_base.cc:1166] Dumped server information to /tmp/dist-test-task1JRg7N/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1781955746769051-7083-0/minicluster-data/master-0/data/info.pb
I20260620 11:46:31.617166 7083 external_mini_cluster.cc:1388] Started /tmp/dist-test-task1JRg7N/build/release/bin/kudu as pid 28023
I20260620 11:46:31.617293 7083 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task1JRg7N/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1781955746769051-7083-0/minicluster-data/master-0/wal/instance
I20260620 11:46:31.617487 7083 external_mini_cluster.cc:1428] Setting key 4be5c45b31647183d1f5b89b60ccadb6
I20260620 11:46:31.617654 28091 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 11:46:31.621117 28091 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 41db0c1d13cf4bf4bed55997733bcbef: Bootstrap starting.
I20260620 11:46:31.621833 28091 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 41db0c1d13cf4bf4bed55997733bcbef: Neither blocks nor log segments found. Creating new log.
I20260620 11:46:31.622139 28091 log.cc:824] T 00000000000000000000000000000000 P 41db0c1d13cf4bf4bed55997733bcbef: Log is configured to *not* fsync() on all Append() calls
I20260620 11:46:31.622979 28091 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 41db0c1d13cf4bf4bed55997733bcbef: No bootstrap required, opened a new log
I20260620 11:46:31.623710 28094 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 11:46:31.618032 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.133:49519 (local address 127.0.0.1:33349)
0620 11:46:31.618235 (+ 203us) server_negotiation.cc:206] Beginning negotiation
0620 11:46:31.618245 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0620 11:46:31.618276 (+ 31us) server_negotiation.cc:407] Connection header received
0620 11:46:31.618908 (+ 632us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 11:46:31.618917 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 11:46:31.618982 (+ 65us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 11:46:31.619135 (+ 153us) server_negotiation.cc:226] Negotiated authn=SASL
0620 11:46:31.619302 (+ 167us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 11:46:31.620651 (+ 1349us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 11:46:31.621483 (+ 832us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 11:46:31.622161 (+ 678us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 11:46:31.622273 (+ 112us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 11:46:31.623035 (+ 762us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 11:46:31.623039 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 11:46:31.623040 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 11:46:31.623244 (+ 204us) server_negotiation.cc:922] calling sasl_server_start()
0620 11:46:31.623293 (+ 49us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 11:46:31.623307 (+ 14us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 11:46:31.623396 (+ 89us) server_negotiation.cc:1035] Waiting for connection context
0620 11:46:31.623474 (+ 78us) server_negotiation.cc:299] Negotiation successful
0620 11:46:31.623501 (+ 27us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":147,"thread_start_us":94,"threads_started":1}
I20260620 11:46:31.624867 7083 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260620 11:46:31.625407 28091 raft_consensus.cc:348] T 00000000000000000000000000000000 P 41db0c1d13cf4bf4bed55997733bcbef [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "41db0c1d13cf4bf4bed55997733bcbef" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 33349 } }
I20260620 11:46:31.625589 28091 raft_consensus.cc:374] T 00000000000000000000000000000000 P 41db0c1d13cf4bf4bed55997733bcbef [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 11:46:31.625641 28091 raft_consensus.cc:729] T 00000000000000000000000000000000 P 41db0c1d13cf4bf4bed55997733bcbef [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 41db0c1d13cf4bf4bed55997733bcbef, State: Initialized, Role: FOLLOWER
I20260620 11:46:31.625798 28091 consensus_queue.cc:260] T 00000000000000000000000000000000 P 41db0c1d13cf4bf4bed55997733bcbef [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "41db0c1d13cf4bf4bed55997733bcbef" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 33349 } }
I20260620 11:46:31.625900 28091 raft_consensus.cc:388] T 00000000000000000000000000000000 P 41db0c1d13cf4bf4bed55997733bcbef [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260620 11:46:31.625955 28091 raft_consensus.cc:482] T 00000000000000000000000000000000 P 41db0c1d13cf4bf4bed55997733bcbef [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260620 11:46:31.626019 28091 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 41db0c1d13cf4bf4bed55997733bcbef [term 0 FOLLOWER]: Advancing to term 1
I20260620 11:46:31.626803 28091 raft_consensus.cc:504] T 00000000000000000000000000000000 P 41db0c1d13cf4bf4bed55997733bcbef [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "41db0c1d13cf4bf4bed55997733bcbef" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 33349 } }
I20260620 11:46:31.626919 28091 leader_election.cc:302] T 00000000000000000000000000000000 P 41db0c1d13cf4bf4bed55997733bcbef [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 41db0c1d13cf4bf4bed55997733bcbef; no voters:
I20260620 11:46:31.627087 28091 leader_election.cc:288] T 00000000000000000000000000000000 P 41db0c1d13cf4bf4bed55997733bcbef [CANDIDATE]: Term 1 election: Requested vote from peers
I20260620 11:46:31.627125 28101 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 41db0c1d13cf4bf4bed55997733bcbef [term 1 FOLLOWER]: Leader election won for term 1
I20260620 11:46:31.627257 28101 raft_consensus.cc:686] T 00000000000000000000000000000000 P 41db0c1d13cf4bf4bed55997733bcbef [term 1 LEADER]: Becoming Leader. State: Replica: 41db0c1d13cf4bf4bed55997733bcbef, State: Running, Role: LEADER
I20260620 11:46:31.627349 28101 consensus_queue.cc:237] T 00000000000000000000000000000000 P 41db0c1d13cf4bf4bed55997733bcbef [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "41db0c1d13cf4bf4bed55997733bcbef" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 33349 } }
I20260620 11:46:31.627353 28091 sys_catalog.cc:564] T 00000000000000000000000000000000 P 41db0c1d13cf4bf4bed55997733bcbef [sys.catalog]: configured and running, proceeding with master startup.
I20260620 11:46:31.627684 28101 sys_catalog.cc:455] T 00000000000000000000000000000000 P 41db0c1d13cf4bf4bed55997733bcbef [sys.catalog]: SysCatalogTable state changed. Reason: New leader 41db0c1d13cf4bf4bed55997733bcbef. Latest consensus state: current_term: 1 leader_uuid: "41db0c1d13cf4bf4bed55997733bcbef" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "41db0c1d13cf4bf4bed55997733bcbef" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 33349 } } }
I20260620 11:46:31.627777 28101 sys_catalog.cc:458] T 00000000000000000000000000000000 P 41db0c1d13cf4bf4bed55997733bcbef [sys.catalog]: This master's current role is: LEADER
I20260620 11:46:31.628024 28107 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260620 11:46:31.628515 28107 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260620 11:46:31.629837 28102 sys_catalog.cc:455] T 00000000000000000000000000000000 P 41db0c1d13cf4bf4bed55997733bcbef [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "41db0c1d13cf4bf4bed55997733bcbef" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "41db0c1d13cf4bf4bed55997733bcbef" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 33349 } } }
I20260620 11:46:31.629925 28102 sys_catalog.cc:458] T 00000000000000000000000000000000 P 41db0c1d13cf4bf4bed55997733bcbef [sys.catalog]: This master's current role is: LEADER
I20260620 11:46:31.630111 28094 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 11:46:31.626394 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.133:52531 (local address 127.0.0.1:33349)
0620 11:46:31.626444 (+ 50us) server_negotiation.cc:206] Beginning negotiation
0620 11:46:31.626446 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 11:46:31.626592 (+ 146us) server_negotiation.cc:407] Connection header received
0620 11:46:31.626645 (+ 53us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 11:46:31.626646 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 11:46:31.626654 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 11:46:31.626699 (+ 45us) server_negotiation.cc:226] Negotiated authn=SASL
0620 11:46:31.626848 (+ 149us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 11:46:31.627572 (+ 724us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 11:46:31.628704 (+ 1132us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 11:46:31.628992 (+ 288us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 11:46:31.629079 (+ 87us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 11:46:31.629771 (+ 692us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 11:46:31.629772 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 11:46:31.629773 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 11:46:31.629777 (+ 4us) server_negotiation.cc:922] calling sasl_server_start()
0620 11:46:31.629791 (+ 14us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 11:46:31.629796 (+ 5us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 11:46:31.629864 (+ 68us) server_negotiation.cc:1035] Waiting for connection context
0620 11:46:31.630034 (+ 170us) server_negotiation.cc:299] Negotiation successful
0620 11:46:31.630045 (+ 11us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":31}
I20260620 11:46:31.630999 28107 catalog_manager.cc:1347] Generated new cluster ID: f059e719bd5646559f2f1025f6ecb032
I20260620 11:46:31.631054 28107 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260620 11:46:31.645064 28107 catalog_manager.cc:1370] Generated new certificate authority record
I20260620 11:46:31.645766 28107 catalog_manager.cc:1504] Loading token signing keys...
I20260620 11:46:31.649341 28107 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 41db0c1d13cf4bf4bed55997733bcbef: Generated new TSK 0
I20260620 11:46:31.649506 28107 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260620 11:46:31.657527 7083 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task1JRg7N/build/release/bin/kudu with pid 28023
W20260620 11:46:31.661790 28008 connection.cc:535] client connection to 127.0.0.1:33349 recv error: Network error: failed to read from TLS socket (remote: 127.0.0.1:33349): Connection reset by peer (error 104)
2026-06-20T11:46:31Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/7 (1318 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/8
Loading random data
Initializing database '/tmp/dist-test-task1JRg7N/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1781955746769051-7083-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 20 11:46:31 dist-test-slave-cjc6 krb5kdc[28122](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 20 11:46:31 dist-test-slave-cjc6 krb5kdc[28122](info): set up 2 sockets
Jun 20 11:46:31 dist-test-slave-cjc6 krb5kdc[28122](info): commencing operation
krb5kdc: starting...
W20260620 11:46:32.701817 7083 mini_kdc.cc:121] Time spent starting KDC: real 1.026s user 0.001s sys 0.004s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task1JRg7N/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1781955746769051-7083-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task1JRg7N/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1781955746769051-7083-0/krb5kdc/test-user.keytab.
Jun 20 11:46:32 dist-test-slave-cjc6 krb5kdc[28122](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781955992, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-20T11:46:32Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-20T11:46:32Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task1JRg7N/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1781955746769051-7083-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task1JRg7N/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1781955746769051-7083-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260620 11:46:32.816375 7083 external_mini_cluster.cc:1326] Running /tmp/dist-test-task1JRg7N/build/release/bin/kudu
/tmp/dist-test-task1JRg7N/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task1JRg7N/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1781955746769051-7083-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task1JRg7N/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1781955746769051-7083-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task1JRg7N/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1781955746769051-7083-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task1JRg7N/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1781955746769051-7083-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:43253
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-task1JRg7N/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1781955746769051-7083-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:44761
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:43253
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=disabled
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-task1JRg7N/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1781955746769051-7083-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task1JRg7N/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1781955746769051-7083-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task1JRg7N/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1781955746769051-7083-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 11:46:32.895172 28135 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 11:46:32.895413 28135 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 11:46:32.895447 28135 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 11:46:32.896559 28135 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 11:46:32.896631 28135 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 11:46:32.896656 28135 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 11:46:32.896682 28135 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 11:46:32.896747 28135 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 11:46:32.898249 28135 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:44761
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task1JRg7N/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1781955746769051-7083-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task1JRg7N/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1781955746769051-7083-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:43253
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task1JRg7N/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1781955746769051-7083-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:43253
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task1JRg7N/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1781955746769051-7083-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.28135
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task1JRg7N/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1781955746769051-7083-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 20 Jun 2026 10:32:37 UTC on ccycloud.quasar-bgyofv.root.comops.site
build id 4457173
I20260620 11:46:32.898525 28135 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 11:46:32.898746 28135 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 11:46:32.903430 28143 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 11:46:32.903452 28140 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 11:46:32.903435 28141 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 11:46:32.903704 28135 server_base.cc:1034] running on GCE node
I20260620 11:46:32.903934 28135 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 11:46:32.904206 28135 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 11:46:32.905364 28135 hybrid_clock.cc:648] HybridClock initialized: now 1781955992905344 us; error 38 us; skew 500 ppm
Jun 20 11:46:32 dist-test-slave-cjc6 krb5kdc[28122](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781955992, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 11:46:32.907814 28135 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260620 11:46:32.908248 28135 webserver.cc:460] Webserver started at http://127.0.0.1:38825/ using document root <none> and password file <none>
I20260620 11:46:32.908438 28135 fs_manager.cc:359] Metadata directory not provided
I20260620 11:46:32.908480 28135 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 11:46:32.908567 28135 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 11:46:32.909535 28135 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task1JRg7N/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1781955746769051-7083-0/minicluster-data/master-0/data/instance:
uuid: "9ca3a78a48674a75a39dfecc854c8abc"
format_stamp: "Formatted at 2026-06-20 11:46:32 on dist-test-slave-cjc6"
server_key: "29c7cbf5274b9a6080fcb80074fdd5a5"
server_key_iv: "e4ad9ecb4cc81e4e73bb45b71bfc553c"
server_key_version: "encryptionkey@0"
I20260620 11:46:32.909921 28135 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task1JRg7N/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1781955746769051-7083-0/minicluster-data/master-0/wal/instance:
uuid: "9ca3a78a48674a75a39dfecc854c8abc"
format_stamp: "Formatted at 2026-06-20 11:46:32 on dist-test-slave-cjc6"
server_key: "29c7cbf5274b9a6080fcb80074fdd5a5"
server_key_iv: "e4ad9ecb4cc81e4e73bb45b71bfc553c"
server_key_version: "encryptionkey@0"
I20260620 11:46:32.911193 28135 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.001s
I20260620 11:46:32.911854 28150 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 11:46:32.912058 28135 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.001s
I20260620 11:46:32.912143 28135 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task1JRg7N/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1781955746769051-7083-0/minicluster-data/master-0/data,/tmp/dist-test-task1JRg7N/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1781955746769051-7083-0/minicluster-data/master-0/wal
uuid: "9ca3a78a48674a75a39dfecc854c8abc"
format_stamp: "Formatted at 2026-06-20 11:46:32 on dist-test-slave-cjc6"
server_key: "29c7cbf5274b9a6080fcb80074fdd5a5"
server_key_iv: "e4ad9ecb4cc81e4e73bb45b71bfc553c"
server_key_version: "encryptionkey@0"
I20260620 11:46:32.912196 28135 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task1JRg7N/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1781955746769051-7083-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task1JRg7N/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1781955746769051-7083-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task1JRg7N/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1781955746769051-7083-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 11:46:32.915361 28135 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 11:46:32.917145 28135 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 11:46:32.917248 28135 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 11:46:32.920946 28135 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:43253
I20260620 11:46:32.920969 28202 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:43253 every 8 connection(s)
I20260620 11:46:32.921300 28135 server_base.cc:1166] Dumped server information to /tmp/dist-test-task1JRg7N/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1781955746769051-7083-0/minicluster-data/master-0/data/info.pb
I20260620 11:46:32.922271 28203 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 11:46:32.924368 28203 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 9ca3a78a48674a75a39dfecc854c8abc: Bootstrap starting.
I20260620 11:46:32.924909 28203 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 9ca3a78a48674a75a39dfecc854c8abc: Neither blocks nor log segments found. Creating new log.
I20260620 11:46:32.925197 28203 log.cc:824] T 00000000000000000000000000000000 P 9ca3a78a48674a75a39dfecc854c8abc: Log is configured to *not* fsync() on all Append() calls
I20260620 11:46:32.925921 28203 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 9ca3a78a48674a75a39dfecc854c8abc: No bootstrap required, opened a new log
I20260620 11:46:32.927433 28203 raft_consensus.cc:348] T 00000000000000000000000000000000 P 9ca3a78a48674a75a39dfecc854c8abc [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9ca3a78a48674a75a39dfecc854c8abc" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 43253 } }
I20260620 11:46:32.927587 28203 raft_consensus.cc:374] T 00000000000000000000000000000000 P 9ca3a78a48674a75a39dfecc854c8abc [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 11:46:32.927634 28203 raft_consensus.cc:729] T 00000000000000000000000000000000 P 9ca3a78a48674a75a39dfecc854c8abc [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9ca3a78a48674a75a39dfecc854c8abc, State: Initialized, Role: FOLLOWER
I20260620 11:46:32.927776 28203 consensus_queue.cc:260] T 00000000000000000000000000000000 P 9ca3a78a48674a75a39dfecc854c8abc [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9ca3a78a48674a75a39dfecc854c8abc" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 43253 } }
I20260620 11:46:32.927857 28203 raft_consensus.cc:388] T 00000000000000000000000000000000 P 9ca3a78a48674a75a39dfecc854c8abc [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260620 11:46:32.927899 28203 raft_consensus.cc:482] T 00000000000000000000000000000000 P 9ca3a78a48674a75a39dfecc854c8abc [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260620 11:46:32.927954 28203 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 9ca3a78a48674a75a39dfecc854c8abc [term 0 FOLLOWER]: Advancing to term 1
I20260620 11:46:32.928470 28203 raft_consensus.cc:504] T 00000000000000000000000000000000 P 9ca3a78a48674a75a39dfecc854c8abc [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9ca3a78a48674a75a39dfecc854c8abc" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 43253 } }
I20260620 11:46:32.928586 28203 leader_election.cc:302] T 00000000000000000000000000000000 P 9ca3a78a48674a75a39dfecc854c8abc [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 9ca3a78a48674a75a39dfecc854c8abc; no voters:
I20260620 11:46:32.928759 28203 leader_election.cc:288] T 00000000000000000000000000000000 P 9ca3a78a48674a75a39dfecc854c8abc [CANDIDATE]: Term 1 election: Requested vote from peers
I20260620 11:46:32.928800 28206 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 9ca3a78a48674a75a39dfecc854c8abc [term 1 FOLLOWER]: Leader election won for term 1
I20260620 11:46:32.928965 28206 raft_consensus.cc:686] T 00000000000000000000000000000000 P 9ca3a78a48674a75a39dfecc854c8abc [term 1 LEADER]: Becoming Leader. State: Replica: 9ca3a78a48674a75a39dfecc854c8abc, State: Running, Role: LEADER
I20260620 11:46:32.929019 28203 sys_catalog.cc:564] T 00000000000000000000000000000000 P 9ca3a78a48674a75a39dfecc854c8abc [sys.catalog]: configured and running, proceeding with master startup.
I20260620 11:46:32.929049 28206 consensus_queue.cc:237] T 00000000000000000000000000000000 P 9ca3a78a48674a75a39dfecc854c8abc [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9ca3a78a48674a75a39dfecc854c8abc" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 43253 } }
I20260620 11:46:32.929401 28208 sys_catalog.cc:455] T 00000000000000000000000000000000 P 9ca3a78a48674a75a39dfecc854c8abc [sys.catalog]: SysCatalogTable state changed. Reason: New leader 9ca3a78a48674a75a39dfecc854c8abc. Latest consensus state: current_term: 1 leader_uuid: "9ca3a78a48674a75a39dfecc854c8abc" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9ca3a78a48674a75a39dfecc854c8abc" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 43253 } } }
I20260620 11:46:32.929494 28208 sys_catalog.cc:458] T 00000000000000000000000000000000 P 9ca3a78a48674a75a39dfecc854c8abc [sys.catalog]: This master's current role is: LEADER
I20260620 11:46:32.929605 28207 sys_catalog.cc:455] T 00000000000000000000000000000000 P 9ca3a78a48674a75a39dfecc854c8abc [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "9ca3a78a48674a75a39dfecc854c8abc" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9ca3a78a48674a75a39dfecc854c8abc" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 43253 } } }
I20260620 11:46:32.929672 28207 sys_catalog.cc:458] T 00000000000000000000000000000000 P 9ca3a78a48674a75a39dfecc854c8abc [sys.catalog]: This master's current role is: LEADER
I20260620 11:46:32.930073 28214 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260620 11:46:32.930511 28214 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260620 11:46:32.930760 7083 external_mini_cluster.cc:1388] Started /tmp/dist-test-task1JRg7N/build/release/bin/kudu as pid 28135
I20260620 11:46:32.930897 7083 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task1JRg7N/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1781955746769051-7083-0/minicluster-data/master-0/wal/instance
I20260620 11:46:32.931105 7083 external_mini_cluster.cc:1428] Setting key 03ede1df0d61b04aaad6922a5ed7ff8f
I20260620 11:46:32.933274 28225 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 11:46:32.931519 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.133:47203 (local address 127.0.0.1:43253)
0620 11:46:32.931679 (+ 160us) server_negotiation.cc:206] Beginning negotiation
0620 11:46:32.931687 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0620 11:46:32.931894 (+ 207us) server_negotiation.cc:407] Connection header received
0620 11:46:32.932027 (+ 133us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 11:46:32.932034 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 11:46:32.932066 (+ 32us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 11:46:32.932225 (+ 159us) server_negotiation.cc:226] Negotiated authn=SASL
0620 11:46:32.932886 (+ 661us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 11:46:32.932890 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 11:46:32.932890 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 11:46:32.932894 (+ 4us) server_negotiation.cc:922] calling sasl_server_start()
0620 11:46:32.932935 (+ 41us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 11:46:32.932950 (+ 15us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 11:46:32.933029 (+ 79us) server_negotiation.cc:1035] Waiting for connection context
0620 11:46:32.933064 (+ 35us) server_negotiation.cc:299] Negotiation successful
0620 11:46:32.933106 (+ 42us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":112,"thread_start_us":68,"threads_started":1}
I20260620 11:46:32.933790 28214 catalog_manager.cc:1347] Generated new cluster ID: 69408459eac34211bed5535ede12517b
I20260620 11:46:32.933841 28214 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260620 11:46:32.934269 7083 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260620 11:46:32.936739 28225 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 11:46:32.935642 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.133:45379 (local address 127.0.0.1:43253)
0620 11:46:32.935677 (+ 35us) server_negotiation.cc:206] Beginning negotiation
0620 11:46:32.935679 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 11:46:32.935846 (+ 167us) server_negotiation.cc:407] Connection header received
0620 11:46:32.935875 (+ 29us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 11:46:32.935876 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 11:46:32.935881 (+ 5us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 11:46:32.935946 (+ 65us) server_negotiation.cc:226] Negotiated authn=SASL
0620 11:46:32.936522 (+ 576us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 11:46:32.936523 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 11:46:32.936523 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 11:46:32.936527 (+ 4us) server_negotiation.cc:922] calling sasl_server_start()
0620 11:46:32.936541 (+ 14us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 11:46:32.936546 (+ 5us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 11:46:32.936608 (+ 62us) server_negotiation.cc:1035] Waiting for connection context
0620 11:46:32.936657 (+ 49us) server_negotiation.cc:299] Negotiation successful
0620 11:46:32.936669 (+ 12us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":20}
I20260620 11:46:32.956918 28214 catalog_manager.cc:1370] Generated new certificate authority record
I20260620 11:46:32.957122 28214 catalog_manager.cc:1504] Loading token signing keys...
I20260620 11:46:32.975287 28214 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 9ca3a78a48674a75a39dfecc854c8abc: Generated new TSK 0
I20260620 11:46:32.975531 28214 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260620 11:46:32.988905 7083 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task1JRg7N/build/release/bin/kudu with pid 28135
2026-06-20T11:46:32Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/8 (1331 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/9
Loading random data
Initializing database '/tmp/dist-test-task1JRg7N/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1781955746769051-7083-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 20 11:46:33 dist-test-slave-cjc6 krb5kdc[28234](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 20 11:46:33 dist-test-slave-cjc6 krb5kdc[28234](info): set up 2 sockets
Jun 20 11:46:33 dist-test-slave-cjc6 krb5kdc[28234](info): commencing operation
krb5kdc: starting...
W20260620 11:46:34.017868 7083 mini_kdc.cc:121] Time spent starting KDC: real 1.011s user 0.002s sys 0.004s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task1JRg7N/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1781955746769051-7083-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task1JRg7N/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1781955746769051-7083-0/krb5kdc/test-user.keytab.
Jun 20 11:46:34 dist-test-slave-cjc6 krb5kdc[28234](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781955994, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-20T11:46:34Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-20T11:46:34Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task1JRg7N/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1781955746769051-7083-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task1JRg7N/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1781955746769051-7083-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260620 11:46:34.129909 7083 external_mini_cluster.cc:1326] Running /tmp/dist-test-task1JRg7N/build/release/bin/kudu
/tmp/dist-test-task1JRg7N/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task1JRg7N/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1781955746769051-7083-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task1JRg7N/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1781955746769051-7083-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task1JRg7N/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1781955746769051-7083-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task1JRg7N/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1781955746769051-7083-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:45057
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-task1JRg7N/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1781955746769051-7083-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:39425
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:45057
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=required
--rpc_encryption=required
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-task1JRg7N/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1781955746769051-7083-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task1JRg7N/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1781955746769051-7083-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task1JRg7N/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1781955746769051-7083-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 11:46:34.209053 28247 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 11:46:34.209272 28247 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 11:46:34.209295 28247 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 11:46:34.210412 28247 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 11:46:34.210451 28247 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 11:46:34.210465 28247 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 11:46:34.210479 28247 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 11:46:34.210490 28247 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 11:46:34.211905 28247 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:39425
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task1JRg7N/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1781955746769051-7083-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task1JRg7N/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1781955746769051-7083-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:45057
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task1JRg7N/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1781955746769051-7083-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:45057
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task1JRg7N/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1781955746769051-7083-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.28247
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task1JRg7N/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1781955746769051-7083-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 20 Jun 2026 10:32:37 UTC on ccycloud.quasar-bgyofv.root.comops.site
build id 4457173
I20260620 11:46:34.212113 28247 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 11:46:34.212286 28247 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260620 11:46:34.217170 28247 server_base.cc:1034] running on GCE node
W20260620 11:46:34.217123 28253 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 11:46:34.217267 28255 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 11:46:34.217141 28252 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 11:46:34.217571 28247 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 11:46:34.217875 28247 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 11:46:34.219025 28247 hybrid_clock.cc:648] HybridClock initialized: now 1781955994219010 us; error 30 us; skew 500 ppm
Jun 20 11:46:34 dist-test-slave-cjc6 krb5kdc[28234](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781955994, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 11:46:34.221477 28247 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260620 11:46:34.221980 28247 webserver.cc:460] Webserver started at http://127.0.0.1:46345/ using document root <none> and password file <none>
I20260620 11:46:34.222208 28247 fs_manager.cc:359] Metadata directory not provided
I20260620 11:46:34.222275 28247 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 11:46:34.222424 28247 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 11:46:34.223461 28247 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task1JRg7N/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1781955746769051-7083-0/minicluster-data/master-0/data/instance:
uuid: "7bea0c0004f341ccb5c2af53b77bf25a"
format_stamp: "Formatted at 2026-06-20 11:46:34 on dist-test-slave-cjc6"
server_key: "a21839ea5579674c1279bdc90e6e6dc0"
server_key_iv: "4566992162431d9e6810f8dc5f42f1f2"
server_key_version: "encryptionkey@0"
I20260620 11:46:34.223836 28247 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task1JRg7N/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1781955746769051-7083-0/minicluster-data/master-0/wal/instance:
uuid: "7bea0c0004f341ccb5c2af53b77bf25a"
format_stamp: "Formatted at 2026-06-20 11:46:34 on dist-test-slave-cjc6"
server_key: "a21839ea5579674c1279bdc90e6e6dc0"
server_key_iv: "4566992162431d9e6810f8dc5f42f1f2"
server_key_version: "encryptionkey@0"
I20260620 11:46:34.225126 28247 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260620 11:46:34.225873 28262 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 11:46:34.226043 28247 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260620 11:46:34.226143 28247 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task1JRg7N/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1781955746769051-7083-0/minicluster-data/master-0/data,/tmp/dist-test-task1JRg7N/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1781955746769051-7083-0/minicluster-data/master-0/wal
uuid: "7bea0c0004f341ccb5c2af53b77bf25a"
format_stamp: "Formatted at 2026-06-20 11:46:34 on dist-test-slave-cjc6"
server_key: "a21839ea5579674c1279bdc90e6e6dc0"
server_key_iv: "4566992162431d9e6810f8dc5f42f1f2"
server_key_version: "encryptionkey@0"
I20260620 11:46:34.226227 28247 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task1JRg7N/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1781955746769051-7083-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task1JRg7N/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1781955746769051-7083-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task1JRg7N/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1781955746769051-7083-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 11:46:34.239153 28247 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 11:46:34.241397 28247 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 11:46:34.241560 28247 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 11:46:34.245651 28247 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:45057
I20260620 11:46:34.245683 28314 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:45057 every 8 connection(s)
I20260620 11:46:34.246079 28247 server_base.cc:1166] Dumped server information to /tmp/dist-test-task1JRg7N/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1781955746769051-7083-0/minicluster-data/master-0/data/info.pb
I20260620 11:46:34.247090 28315 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 11:46:34.249193 28315 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 7bea0c0004f341ccb5c2af53b77bf25a: Bootstrap starting.
I20260620 11:46:34.249806 28315 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 7bea0c0004f341ccb5c2af53b77bf25a: Neither blocks nor log segments found. Creating new log.
I20260620 11:46:34.250053 28315 log.cc:824] T 00000000000000000000000000000000 P 7bea0c0004f341ccb5c2af53b77bf25a: Log is configured to *not* fsync() on all Append() calls
I20260620 11:46:34.250667 28315 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 7bea0c0004f341ccb5c2af53b77bf25a: No bootstrap required, opened a new log
I20260620 11:46:34.252233 28315 raft_consensus.cc:348] T 00000000000000000000000000000000 P 7bea0c0004f341ccb5c2af53b77bf25a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7bea0c0004f341ccb5c2af53b77bf25a" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 45057 } }
I20260620 11:46:34.252362 28315 raft_consensus.cc:374] T 00000000000000000000000000000000 P 7bea0c0004f341ccb5c2af53b77bf25a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 11:46:34.252385 28315 raft_consensus.cc:729] T 00000000000000000000000000000000 P 7bea0c0004f341ccb5c2af53b77bf25a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7bea0c0004f341ccb5c2af53b77bf25a, State: Initialized, Role: FOLLOWER
I20260620 11:46:34.252490 28315 consensus_queue.cc:260] T 00000000000000000000000000000000 P 7bea0c0004f341ccb5c2af53b77bf25a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7bea0c0004f341ccb5c2af53b77bf25a" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 45057 } }
I20260620 11:46:34.252539 28315 raft_consensus.cc:388] T 00000000000000000000000000000000 P 7bea0c0004f341ccb5c2af53b77bf25a [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260620 11:46:34.252558 28315 raft_consensus.cc:482] T 00000000000000000000000000000000 P 7bea0c0004f341ccb5c2af53b77bf25a [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260620 11:46:34.252584 28315 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 7bea0c0004f341ccb5c2af53b77bf25a [term 0 FOLLOWER]: Advancing to term 1
I20260620 11:46:34.253032 28315 raft_consensus.cc:504] T 00000000000000000000000000000000 P 7bea0c0004f341ccb5c2af53b77bf25a [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7bea0c0004f341ccb5c2af53b77bf25a" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 45057 } }
I20260620 11:46:34.253110 28315 leader_election.cc:302] T 00000000000000000000000000000000 P 7bea0c0004f341ccb5c2af53b77bf25a [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 7bea0c0004f341ccb5c2af53b77bf25a; no voters:
I20260620 11:46:34.253230 28315 leader_election.cc:288] T 00000000000000000000000000000000 P 7bea0c0004f341ccb5c2af53b77bf25a [CANDIDATE]: Term 1 election: Requested vote from peers
I20260620 11:46:34.253304 28318 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 7bea0c0004f341ccb5c2af53b77bf25a [term 1 FOLLOWER]: Leader election won for term 1
I20260620 11:46:34.253484 28318 raft_consensus.cc:686] T 00000000000000000000000000000000 P 7bea0c0004f341ccb5c2af53b77bf25a [term 1 LEADER]: Becoming Leader. State: Replica: 7bea0c0004f341ccb5c2af53b77bf25a, State: Running, Role: LEADER
I20260620 11:46:34.253497 28315 sys_catalog.cc:564] T 00000000000000000000000000000000 P 7bea0c0004f341ccb5c2af53b77bf25a [sys.catalog]: configured and running, proceeding with master startup.
I20260620 11:46:34.253571 28318 consensus_queue.cc:237] T 00000000000000000000000000000000 P 7bea0c0004f341ccb5c2af53b77bf25a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7bea0c0004f341ccb5c2af53b77bf25a" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 45057 } }
I20260620 11:46:34.253959 28320 sys_catalog.cc:455] T 00000000000000000000000000000000 P 7bea0c0004f341ccb5c2af53b77bf25a [sys.catalog]: SysCatalogTable state changed. Reason: New leader 7bea0c0004f341ccb5c2af53b77bf25a. Latest consensus state: current_term: 1 leader_uuid: "7bea0c0004f341ccb5c2af53b77bf25a" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7bea0c0004f341ccb5c2af53b77bf25a" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 45057 } } }
I20260620 11:46:34.254050 28320 sys_catalog.cc:458] T 00000000000000000000000000000000 P 7bea0c0004f341ccb5c2af53b77bf25a [sys.catalog]: This master's current role is: LEADER
I20260620 11:46:34.254091 28319 sys_catalog.cc:455] T 00000000000000000000000000000000 P 7bea0c0004f341ccb5c2af53b77bf25a [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "7bea0c0004f341ccb5c2af53b77bf25a" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7bea0c0004f341ccb5c2af53b77bf25a" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 45057 } } }
I20260620 11:46:34.254158 28319 sys_catalog.cc:458] T 00000000000000000000000000000000 P 7bea0c0004f341ccb5c2af53b77bf25a [sys.catalog]: This master's current role is: LEADER
I20260620 11:46:34.254251 7083 external_mini_cluster.cc:1388] Started /tmp/dist-test-task1JRg7N/build/release/bin/kudu as pid 28247
I20260620 11:46:34.254369 7083 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task1JRg7N/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1781955746769051-7083-0/minicluster-data/master-0/wal/instance
I20260620 11:46:34.254580 7083 external_mini_cluster.cc:1428] Setting key 883213c07f534d66385397e3244447ea
I20260620 11:46:34.254913 28324 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260620 11:46:34.255437 28324 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260620 11:46:34.257787 28324 catalog_manager.cc:1347] Generated new cluster ID: e62dd14d971c4b378bc3a600ffa6a5c4
I20260620 11:46:34.257860 28324 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
Jun 20 11:46:34 dist-test-slave-cjc6 krb5kdc[28234](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781955994, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.0.1@KRBTEST.COM
I20260620 11:46:34.266206 28331 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 11:46:34.255053 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.133:42711 (local address 127.0.0.1:45057)
0620 11:46:34.255225 (+ 172us) server_negotiation.cc:206] Beginning negotiation
0620 11:46:34.255236 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0620 11:46:34.255345 (+ 109us) server_negotiation.cc:407] Connection header received
0620 11:46:34.255508 (+ 163us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 11:46:34.255516 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 11:46:34.255593 (+ 77us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 11:46:34.255753 (+ 160us) server_negotiation.cc:226] Negotiated authn=SASL
0620 11:46:34.256531 (+ 778us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 11:46:34.257928 (+ 1397us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 11:46:34.258974 (+ 1046us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 11:46:34.259613 (+ 639us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 11:46:34.259728 (+ 115us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 11:46:34.262430 (+ 2702us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 11:46:34.262435 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 11:46:34.262435 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 11:46:34.262446 (+ 11us) server_negotiation.cc:922] calling sasl_server_start()
0620 11:46:34.264462 (+ 2016us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 11:46:34.264917 (+ 455us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 11:46:34.264920 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 11:46:34.264920 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 11:46:34.265037 (+ 117us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 11:46:34.265281 (+ 244us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 11:46:34.265281 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 11:46:34.265282 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 11:46:34.265568 (+ 286us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 11:46:34.265735 (+ 167us) server_negotiation.cc:1035] Waiting for connection context
0620 11:46:34.265891 (+ 156us) server_negotiation.cc:299] Negotiation successful
0620 11:46:34.265991 (+ 100us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":119,"thread_start_us":66,"threads_started":1}
I20260620 11:46:34.267277 7083 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260620 11:46:34.269639 28324 catalog_manager.cc:1370] Generated new certificate authority record
I20260620 11:46:34.270146 28324 catalog_manager.cc:1504] Loading token signing keys...
I20260620 11:46:34.275972 28331 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 11:46:34.268669 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.133:42689 (local address 127.0.0.1:45057)
0620 11:46:34.268702 (+ 33us) server_negotiation.cc:206] Beginning negotiation
0620 11:46:34.268704 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 11:46:34.268800 (+ 96us) server_negotiation.cc:407] Connection header received
0620 11:46:34.268875 (+ 75us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 11:46:34.268876 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 11:46:34.268887 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 11:46:34.268936 (+ 49us) server_negotiation.cc:226] Negotiated authn=SASL
0620 11:46:34.269637 (+ 701us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 11:46:34.270422 (+ 785us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 11:46:34.271387 (+ 965us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 11:46:34.271691 (+ 304us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 11:46:34.271760 (+ 69us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 11:46:34.273175 (+ 1415us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 11:46:34.273177 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 11:46:34.273177 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 11:46:34.273184 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0620 11:46:34.274777 (+ 1593us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 11:46:34.275179 (+ 402us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 11:46:34.275180 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 11:46:34.275180 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 11:46:34.275242 (+ 62us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 11:46:34.275460 (+ 218us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 11:46:34.275461 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 11:46:34.275461 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 11:46:34.275581 (+ 120us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 11:46:34.275679 (+ 98us) server_negotiation.cc:1035] Waiting for connection context
0620 11:46:34.275824 (+ 145us) server_negotiation.cc:299] Negotiation successful
0620 11:46:34.275887 (+ 63us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":16}
I20260620 11:46:34.276655 28324 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 7bea0c0004f341ccb5c2af53b77bf25a: Generated new TSK 0
I20260620 11:46:34.276849 28324 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260620 11:46:34.280473 7083 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task1JRg7N/build/release/bin/kudu with pid 28247
W20260620 11:46:34.284562 28232 connection.cc:535] client connection to 127.0.0.1:45057 recv error: Network error: failed to read from TLS socket (remote: 127.0.0.1:45057): Connection reset by peer (error 104)
2026-06-20T11:46:34Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/9 (1291 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/10
Loading random data
Initializing database '/tmp/dist-test-task1JRg7N/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1781955746769051-7083-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 20 11:46:34 dist-test-slave-cjc6 krb5kdc[28346](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 20 11:46:34 dist-test-slave-cjc6 krb5kdc[28346](info): set up 2 sockets
Jun 20 11:46:34 dist-test-slave-cjc6 krb5kdc[28346](info): commencing operation
krb5kdc: starting...
W20260620 11:46:35.313967 7083 mini_kdc.cc:121] Time spent starting KDC: real 1.016s user 0.000s sys 0.006s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task1JRg7N/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1781955746769051-7083-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task1JRg7N/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1781955746769051-7083-0/krb5kdc/test-user.keytab.
Jun 20 11:46:35 dist-test-slave-cjc6 krb5kdc[28346](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781955995, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-20T11:46:35Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-20T11:46:35Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task1JRg7N/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1781955746769051-7083-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task1JRg7N/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1781955746769051-7083-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260620 11:46:35.430018 7083 external_mini_cluster.cc:1326] Running /tmp/dist-test-task1JRg7N/build/release/bin/kudu
/tmp/dist-test-task1JRg7N/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task1JRg7N/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1781955746769051-7083-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task1JRg7N/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1781955746769051-7083-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task1JRg7N/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1781955746769051-7083-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task1JRg7N/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1781955746769051-7083-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:36375
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-task1JRg7N/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1781955746769051-7083-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:40673
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:36375
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=required
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-task1JRg7N/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1781955746769051-7083-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task1JRg7N/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1781955746769051-7083-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task1JRg7N/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1781955746769051-7083-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 11:46:35.511018 28359 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 11:46:35.511231 28359 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 11:46:35.511255 28359 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 11:46:35.512426 28359 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 11:46:35.512473 28359 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 11:46:35.512487 28359 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 11:46:35.512499 28359 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 11:46:35.512511 28359 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 11:46:35.514014 28359 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:40673
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task1JRg7N/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1781955746769051-7083-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task1JRg7N/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1781955746769051-7083-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:36375
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task1JRg7N/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1781955746769051-7083-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:36375
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task1JRg7N/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1781955746769051-7083-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.28359
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task1JRg7N/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1781955746769051-7083-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 20 Jun 2026 10:32:37 UTC on ccycloud.quasar-bgyofv.root.comops.site
build id 4457173
I20260620 11:46:35.514276 28359 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 11:46:35.514514 28359 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 11:46:35.519526 28365 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 11:46:35.519538 28367 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 11:46:35.519521 28364 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 11:46:35.519594 28359 server_base.cc:1034] running on GCE node
I20260620 11:46:35.519905 28359 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 11:46:35.520144 28359 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 11:46:35.521293 28359 hybrid_clock.cc:648] HybridClock initialized: now 1781955995521281 us; error 27 us; skew 500 ppm
Jun 20 11:46:35 dist-test-slave-cjc6 krb5kdc[28346](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781955995, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 11:46:35.523841 28359 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260620 11:46:35.524322 28359 webserver.cc:460] Webserver started at http://127.0.0.1:45203/ using document root <none> and password file <none>
I20260620 11:46:35.524539 28359 fs_manager.cc:359] Metadata directory not provided
I20260620 11:46:35.524583 28359 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 11:46:35.524757 28359 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 11:46:35.525890 28359 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task1JRg7N/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1781955746769051-7083-0/minicluster-data/master-0/data/instance:
uuid: "39a1af30e4fa476da6b732108be4e7af"
format_stamp: "Formatted at 2026-06-20 11:46:35 on dist-test-slave-cjc6"
server_key: "bb48eb867c6bec48c2c31d3fb8a96db1"
server_key_iv: "bf123b1d027cb34546b7ad1da5190375"
server_key_version: "encryptionkey@0"
I20260620 11:46:35.526242 28359 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task1JRg7N/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1781955746769051-7083-0/minicluster-data/master-0/wal/instance:
uuid: "39a1af30e4fa476da6b732108be4e7af"
format_stamp: "Formatted at 2026-06-20 11:46:35 on dist-test-slave-cjc6"
server_key: "bb48eb867c6bec48c2c31d3fb8a96db1"
server_key_iv: "bf123b1d027cb34546b7ad1da5190375"
server_key_version: "encryptionkey@0"
I20260620 11:46:35.527572 28359 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.002s
I20260620 11:46:35.528290 28374 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 11:46:35.528457 28359 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.000s
I20260620 11:46:35.528565 28359 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task1JRg7N/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1781955746769051-7083-0/minicluster-data/master-0/data,/tmp/dist-test-task1JRg7N/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1781955746769051-7083-0/minicluster-data/master-0/wal
uuid: "39a1af30e4fa476da6b732108be4e7af"
format_stamp: "Formatted at 2026-06-20 11:46:35 on dist-test-slave-cjc6"
server_key: "bb48eb867c6bec48c2c31d3fb8a96db1"
server_key_iv: "bf123b1d027cb34546b7ad1da5190375"
server_key_version: "encryptionkey@0"
I20260620 11:46:35.528640 28359 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task1JRg7N/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1781955746769051-7083-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task1JRg7N/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1781955746769051-7083-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task1JRg7N/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1781955746769051-7083-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 11:46:35.551692 28359 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 11:46:35.554163 28359 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 11:46:35.554338 28359 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 11:46:35.558352 28359 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:36375
I20260620 11:46:35.558389 28426 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:36375 every 8 connection(s)
I20260620 11:46:35.558785 28359 server_base.cc:1166] Dumped server information to /tmp/dist-test-task1JRg7N/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1781955746769051-7083-0/minicluster-data/master-0/data/info.pb
I20260620 11:46:35.559662 28427 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 11:46:35.562000 28427 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 39a1af30e4fa476da6b732108be4e7af: Bootstrap starting.
I20260620 11:46:35.562564 28427 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 39a1af30e4fa476da6b732108be4e7af: Neither blocks nor log segments found. Creating new log.
I20260620 11:46:35.562849 28427 log.cc:824] T 00000000000000000000000000000000 P 39a1af30e4fa476da6b732108be4e7af: Log is configured to *not* fsync() on all Append() calls
I20260620 11:46:35.563534 28427 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 39a1af30e4fa476da6b732108be4e7af: No bootstrap required, opened a new log
I20260620 11:46:35.564843 7083 external_mini_cluster.cc:1388] Started /tmp/dist-test-task1JRg7N/build/release/bin/kudu as pid 28359
I20260620 11:46:35.564967 7083 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task1JRg7N/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1781955746769051-7083-0/minicluster-data/master-0/wal/instance
I20260620 11:46:35.565171 7083 external_mini_cluster.cc:1428] Setting key 9162c1ac5641c662e8e937159283479b
I20260620 11:46:35.565529 28427 raft_consensus.cc:348] T 00000000000000000000000000000000 P 39a1af30e4fa476da6b732108be4e7af [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "39a1af30e4fa476da6b732108be4e7af" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 36375 } }
I20260620 11:46:35.565783 28427 raft_consensus.cc:374] T 00000000000000000000000000000000 P 39a1af30e4fa476da6b732108be4e7af [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 11:46:35.565888 28427 raft_consensus.cc:729] T 00000000000000000000000000000000 P 39a1af30e4fa476da6b732108be4e7af [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 39a1af30e4fa476da6b732108be4e7af, State: Initialized, Role: FOLLOWER
I20260620 11:46:35.566037 28427 consensus_queue.cc:260] T 00000000000000000000000000000000 P 39a1af30e4fa476da6b732108be4e7af [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "39a1af30e4fa476da6b732108be4e7af" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 36375 } }
I20260620 11:46:35.566123 28427 raft_consensus.cc:388] T 00000000000000000000000000000000 P 39a1af30e4fa476da6b732108be4e7af [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260620 11:46:35.566147 28427 raft_consensus.cc:482] T 00000000000000000000000000000000 P 39a1af30e4fa476da6b732108be4e7af [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260620 11:46:35.566212 28427 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 39a1af30e4fa476da6b732108be4e7af [term 0 FOLLOWER]: Advancing to term 1
I20260620 11:46:35.567080 28427 raft_consensus.cc:504] T 00000000000000000000000000000000 P 39a1af30e4fa476da6b732108be4e7af [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "39a1af30e4fa476da6b732108be4e7af" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 36375 } }
I20260620 11:46:35.567163 28427 leader_election.cc:302] T 00000000000000000000000000000000 P 39a1af30e4fa476da6b732108be4e7af [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 39a1af30e4fa476da6b732108be4e7af; no voters:
I20260620 11:46:35.567356 28427 leader_election.cc:288] T 00000000000000000000000000000000 P 39a1af30e4fa476da6b732108be4e7af [CANDIDATE]: Term 1 election: Requested vote from peers
I20260620 11:46:35.567377 28432 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 39a1af30e4fa476da6b732108be4e7af [term 1 FOLLOWER]: Leader election won for term 1
I20260620 11:46:35.567497 28432 raft_consensus.cc:686] T 00000000000000000000000000000000 P 39a1af30e4fa476da6b732108be4e7af [term 1 LEADER]: Becoming Leader. State: Replica: 39a1af30e4fa476da6b732108be4e7af, State: Running, Role: LEADER
I20260620 11:46:35.567621 28427 sys_catalog.cc:564] T 00000000000000000000000000000000 P 39a1af30e4fa476da6b732108be4e7af [sys.catalog]: configured and running, proceeding with master startup.
I20260620 11:46:35.567631 28432 consensus_queue.cc:237] T 00000000000000000000000000000000 P 39a1af30e4fa476da6b732108be4e7af [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "39a1af30e4fa476da6b732108be4e7af" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 36375 } }
I20260620 11:46:35.568058 28434 sys_catalog.cc:455] T 00000000000000000000000000000000 P 39a1af30e4fa476da6b732108be4e7af [sys.catalog]: SysCatalogTable state changed. Reason: New leader 39a1af30e4fa476da6b732108be4e7af. Latest consensus state: current_term: 1 leader_uuid: "39a1af30e4fa476da6b732108be4e7af" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "39a1af30e4fa476da6b732108be4e7af" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 36375 } } }
I20260620 11:46:35.568182 28434 sys_catalog.cc:458] T 00000000000000000000000000000000 P 39a1af30e4fa476da6b732108be4e7af [sys.catalog]: This master's current role is: LEADER
I20260620 11:46:35.568384 28433 sys_catalog.cc:455] T 00000000000000000000000000000000 P 39a1af30e4fa476da6b732108be4e7af [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "39a1af30e4fa476da6b732108be4e7af" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "39a1af30e4fa476da6b732108be4e7af" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 36375 } } }
I20260620 11:46:35.568481 28433 sys_catalog.cc:458] T 00000000000000000000000000000000 P 39a1af30e4fa476da6b732108be4e7af [sys.catalog]: This master's current role is: LEADER
W20260620 11:46:35.569492 28448 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 39a1af30e4fa476da6b732108be4e7af: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260620 11:46:35.569581 28448 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260620 11:46:35.569641 28440 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260620 11:46:35.569814 28440 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260620 11:46:35.570839 28431 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 11:46:35.565683 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.133:48191 (local address 127.0.0.1:36375)
0620 11:46:35.565902 (+ 219us) server_negotiation.cc:206] Beginning negotiation
0620 11:46:35.565911 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0620 11:46:35.565970 (+ 59us) server_negotiation.cc:407] Connection header received
0620 11:46:35.566085 (+ 115us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 11:46:35.566091 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 11:46:35.566160 (+ 69us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 11:46:35.566294 (+ 134us) server_negotiation.cc:226] Negotiated authn=SASL
0620 11:46:35.566509 (+ 215us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 11:46:35.567567 (+ 1058us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 11:46:35.568737 (+ 1170us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 11:46:35.569296 (+ 559us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 11:46:35.569413 (+ 117us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 11:46:35.570170 (+ 757us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 11:46:35.570174 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 11:46:35.570174 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 11:46:35.570368 (+ 194us) server_negotiation.cc:922] calling sasl_server_start()
0620 11:46:35.570397 (+ 29us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 11:46:35.570408 (+ 11us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 11:46:35.570501 (+ 93us) server_negotiation.cc:1035] Waiting for connection context
0620 11:46:35.570578 (+ 77us) server_negotiation.cc:299] Negotiation successful
0620 11:46:35.570605 (+ 27us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":146,"thread_start_us":96,"threads_started":1}
I20260620 11:46:35.571501 28440 catalog_manager.cc:1347] Generated new cluster ID: 524b01d9ad124d67ab22f006deab6aab
I20260620 11:46:35.571573 28440 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260620 11:46:35.582472 7083 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260620 11:46:35.587707 28431 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 11:46:35.584061 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.133:60167 (local address 127.0.0.1:36375)
0620 11:46:35.584115 (+ 54us) server_negotiation.cc:206] Beginning negotiation
0620 11:46:35.584118 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0620 11:46:35.584227 (+ 109us) server_negotiation.cc:407] Connection header received
0620 11:46:35.584285 (+ 58us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 11:46:35.584285 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 11:46:35.584297 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 11:46:35.584365 (+ 68us) server_negotiation.cc:226] Negotiated authn=SASL
0620 11:46:35.584520 (+ 155us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 11:46:35.585254 (+ 734us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 11:46:35.586353 (+ 1099us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 11:46:35.586653 (+ 300us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 11:46:35.586719 (+ 66us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 11:46:35.587441 (+ 722us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 11:46:35.587443 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 11:46:35.587443 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 11:46:35.587447 (+ 4us) server_negotiation.cc:922] calling sasl_server_start()
0620 11:46:35.587462 (+ 15us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 11:46:35.587466 (+ 4us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 11:46:35.587515 (+ 49us) server_negotiation.cc:1035] Waiting for connection context
0620 11:46:35.587614 (+ 99us) server_negotiation.cc:299] Negotiation successful
0620 11:46:35.587627 (+ 13us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":29}
I20260620 11:46:35.591809 28440 catalog_manager.cc:1370] Generated new certificate authority record
I20260620 11:46:35.592368 28440 catalog_manager.cc:1504] Loading token signing keys...
I20260620 11:46:35.598016 28440 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 39a1af30e4fa476da6b732108be4e7af: Generated new TSK 0
I20260620 11:46:35.598203 28440 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260620 11:46:35.602480 7083 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task1JRg7N/build/release/bin/kudu with pid 28359
W20260620 11:46:35.607198 28344 connection.cc:535] client connection to 127.0.0.1:36375 recv error: Network error: failed to read from TLS socket (remote: 127.0.0.1:36375): Connection reset by peer (error 104)
2026-06-20T11:46:35Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/10 (1322 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/11
Loading random data
Initializing database '/tmp/dist-test-task1JRg7N/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1781955746769051-7083-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 20 11:46:35 dist-test-slave-cjc6 krb5kdc[28458](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 20 11:46:35 dist-test-slave-cjc6 krb5kdc[28458](info): set up 2 sockets
Jun 20 11:46:35 dist-test-slave-cjc6 krb5kdc[28458](info): commencing operation
krb5kdc: starting...
W20260620 11:46:36.641882 7083 mini_kdc.cc:121] Time spent starting KDC: real 1.021s user 0.000s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task1JRg7N/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1781955746769051-7083-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task1JRg7N/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1781955746769051-7083-0/krb5kdc/test-user.keytab.
Jun 20 11:46:36 dist-test-slave-cjc6 krb5kdc[28458](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781955996, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-20T11:46:36Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-20T11:46:36Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task1JRg7N/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1781955746769051-7083-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task1JRg7N/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1781955746769051-7083-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260620 11:46:36.757907 7083 external_mini_cluster.cc:1326] Running /tmp/dist-test-task1JRg7N/build/release/bin/kudu
/tmp/dist-test-task1JRg7N/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task1JRg7N/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1781955746769051-7083-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task1JRg7N/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1781955746769051-7083-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task1JRg7N/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1781955746769051-7083-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task1JRg7N/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1781955746769051-7083-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:46303
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-task1JRg7N/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1781955746769051-7083-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:36435
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:46303
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=disabled
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-task1JRg7N/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1781955746769051-7083-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task1JRg7N/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1781955746769051-7083-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task1JRg7N/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1781955746769051-7083-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 11:46:36.837726 28471 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 11:46:36.837936 28471 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 11:46:36.837961 28471 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 11:46:36.839053 28471 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 11:46:36.839088 28471 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 11:46:36.839102 28471 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 11:46:36.839115 28471 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 11:46:36.839128 28471 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 11:46:36.840525 28471 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:36435
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task1JRg7N/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1781955746769051-7083-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task1JRg7N/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1781955746769051-7083-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:46303
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task1JRg7N/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1781955746769051-7083-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:46303
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task1JRg7N/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1781955746769051-7083-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.28471
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task1JRg7N/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1781955746769051-7083-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 20 Jun 2026 10:32:37 UTC on ccycloud.quasar-bgyofv.root.comops.site
build id 4457173
I20260620 11:46:36.840723 28471 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 11:46:36.840895 28471 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 11:46:36.845556 28477 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 11:46:36.845732 28471 server_base.cc:1034] running on GCE node
W20260620 11:46:36.845575 28479 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 11:46:36.845616 28476 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 11:46:36.846040 28471 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 11:46:36.846283 28471 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 11:46:36.847424 28471 hybrid_clock.cc:648] HybridClock initialized: now 1781955996847410 us; error 28 us; skew 500 ppm
Jun 20 11:46:36 dist-test-slave-cjc6 krb5kdc[28458](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781955996, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 11:46:36.852609 28471 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260620 11:46:36.853104 28471 webserver.cc:460] Webserver started at http://127.0.0.1:39963/ using document root <none> and password file <none>
I20260620 11:46:36.853353 28471 fs_manager.cc:359] Metadata directory not provided
I20260620 11:46:36.853397 28471 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 11:46:36.853564 28471 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 11:46:36.854713 28471 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task1JRg7N/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1781955746769051-7083-0/minicluster-data/master-0/data/instance:
uuid: "2e39f68bd93a4493af7bfd27b8762366"
format_stamp: "Formatted at 2026-06-20 11:46:36 on dist-test-slave-cjc6"
server_key: "7284cfcbe078e9c8f19c60da41d045c8"
server_key_iv: "72307596e6dc4bb52e40790a0d5ea382"
server_key_version: "encryptionkey@0"
I20260620 11:46:36.855082 28471 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task1JRg7N/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1781955746769051-7083-0/minicluster-data/master-0/wal/instance:
uuid: "2e39f68bd93a4493af7bfd27b8762366"
format_stamp: "Formatted at 2026-06-20 11:46:36 on dist-test-slave-cjc6"
server_key: "7284cfcbe078e9c8f19c60da41d045c8"
server_key_iv: "72307596e6dc4bb52e40790a0d5ea382"
server_key_version: "encryptionkey@0"
I20260620 11:46:36.856384 28471 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260620 11:46:36.857129 28486 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 11:46:36.857342 28471 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260620 11:46:36.857452 28471 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task1JRg7N/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1781955746769051-7083-0/minicluster-data/master-0/data,/tmp/dist-test-task1JRg7N/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1781955746769051-7083-0/minicluster-data/master-0/wal
uuid: "2e39f68bd93a4493af7bfd27b8762366"
format_stamp: "Formatted at 2026-06-20 11:46:36 on dist-test-slave-cjc6"
server_key: "7284cfcbe078e9c8f19c60da41d045c8"
server_key_iv: "72307596e6dc4bb52e40790a0d5ea382"
server_key_version: "encryptionkey@0"
I20260620 11:46:36.857538 28471 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task1JRg7N/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1781955746769051-7083-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task1JRg7N/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1781955746769051-7083-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task1JRg7N/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1781955746769051-7083-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 11:46:36.860874 28471 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 11:46:36.862883 28471 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 11:46:36.863044 28471 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 11:46:36.867066 28471 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:46303
I20260620 11:46:36.867091 28538 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:46303 every 8 connection(s)
I20260620 11:46:36.867422 28471 server_base.cc:1166] Dumped server information to /tmp/dist-test-task1JRg7N/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1781955746769051-7083-0/minicluster-data/master-0/data/info.pb
I20260620 11:46:36.868355 28539 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 11:46:36.870630 28539 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 2e39f68bd93a4493af7bfd27b8762366: Bootstrap starting.
I20260620 11:46:36.871206 28539 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 2e39f68bd93a4493af7bfd27b8762366: Neither blocks nor log segments found. Creating new log.
I20260620 11:46:36.871490 28539 log.cc:824] T 00000000000000000000000000000000 P 2e39f68bd93a4493af7bfd27b8762366: Log is configured to *not* fsync() on all Append() calls
I20260620 11:46:36.872184 7083 external_mini_cluster.cc:1388] Started /tmp/dist-test-task1JRg7N/build/release/bin/kudu as pid 28471
I20260620 11:46:36.872313 28539 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 2e39f68bd93a4493af7bfd27b8762366: No bootstrap required, opened a new log
I20260620 11:46:36.872329 7083 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task1JRg7N/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1781955746769051-7083-0/minicluster-data/master-0/wal/instance
I20260620 11:46:36.872532 7083 external_mini_cluster.cc:1428] Setting key 58aee5e1ca52c3e2dbb64af06bfa6fe2
I20260620 11:46:36.874785 28539 raft_consensus.cc:348] T 00000000000000000000000000000000 P 2e39f68bd93a4493af7bfd27b8762366 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2e39f68bd93a4493af7bfd27b8762366" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 46303 } }
I20260620 11:46:36.874934 28539 raft_consensus.cc:374] T 00000000000000000000000000000000 P 2e39f68bd93a4493af7bfd27b8762366 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 11:46:36.874958 28539 raft_consensus.cc:729] T 00000000000000000000000000000000 P 2e39f68bd93a4493af7bfd27b8762366 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2e39f68bd93a4493af7bfd27b8762366, State: Initialized, Role: FOLLOWER
I20260620 11:46:36.875092 28539 consensus_queue.cc:260] T 00000000000000000000000000000000 P 2e39f68bd93a4493af7bfd27b8762366 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2e39f68bd93a4493af7bfd27b8762366" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 46303 } }
I20260620 11:46:36.875170 28539 raft_consensus.cc:388] T 00000000000000000000000000000000 P 2e39f68bd93a4493af7bfd27b8762366 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260620 11:46:36.875192 28539 raft_consensus.cc:482] T 00000000000000000000000000000000 P 2e39f68bd93a4493af7bfd27b8762366 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260620 11:46:36.875236 28543 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 11:46:36.873067 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.133:37495 (local address 127.0.0.1:46303)
0620 11:46:36.873245 (+ 178us) server_negotiation.cc:206] Beginning negotiation
0620 11:46:36.873253 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0620 11:46:36.873278 (+ 25us) server_negotiation.cc:407] Connection header received
0620 11:46:36.873688 (+ 410us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 11:46:36.873751 (+ 63us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 11:46:36.873786 (+ 35us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 11:46:36.873901 (+ 115us) server_negotiation.cc:226] Negotiated authn=SASL
0620 11:46:36.874505 (+ 604us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 11:46:36.874508 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 11:46:36.874509 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 11:46:36.874512 (+ 3us) server_negotiation.cc:922] calling sasl_server_start()
0620 11:46:36.874546 (+ 34us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 11:46:36.874556 (+ 10us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 11:46:36.874609 (+ 53us) server_negotiation.cc:1035] Waiting for connection context
0620 11:46:36.874697 (+ 88us) server_negotiation.cc:299] Negotiation successful
0620 11:46:36.874798 (+ 101us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":119,"thread_start_us":88,"threads_started":1}
I20260620 11:46:36.875255 28539 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 2e39f68bd93a4493af7bfd27b8762366 [term 0 FOLLOWER]: Advancing to term 1
I20260620 11:46:36.875873 28539 raft_consensus.cc:504] T 00000000000000000000000000000000 P 2e39f68bd93a4493af7bfd27b8762366 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2e39f68bd93a4493af7bfd27b8762366" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 46303 } }
I20260620 11:46:36.875995 28539 leader_election.cc:302] T 00000000000000000000000000000000 P 2e39f68bd93a4493af7bfd27b8762366 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 2e39f68bd93a4493af7bfd27b8762366; no voters:
I20260620 11:46:36.876155 28539 leader_election.cc:288] T 00000000000000000000000000000000 P 2e39f68bd93a4493af7bfd27b8762366 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260620 11:46:36.876148 7083 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260620 11:46:36.876207 28544 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 2e39f68bd93a4493af7bfd27b8762366 [term 1 FOLLOWER]: Leader election won for term 1
I20260620 11:46:36.876368 28544 raft_consensus.cc:686] T 00000000000000000000000000000000 P 2e39f68bd93a4493af7bfd27b8762366 [term 1 LEADER]: Becoming Leader. State: Replica: 2e39f68bd93a4493af7bfd27b8762366, State: Running, Role: LEADER
I20260620 11:46:36.876415 28539 sys_catalog.cc:564] T 00000000000000000000000000000000 P 2e39f68bd93a4493af7bfd27b8762366 [sys.catalog]: configured and running, proceeding with master startup.
I20260620 11:46:36.876516 28544 consensus_queue.cc:237] T 00000000000000000000000000000000 P 2e39f68bd93a4493af7bfd27b8762366 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2e39f68bd93a4493af7bfd27b8762366" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 46303 } }
I20260620 11:46:36.876900 28545 sys_catalog.cc:455] T 00000000000000000000000000000000 P 2e39f68bd93a4493af7bfd27b8762366 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "2e39f68bd93a4493af7bfd27b8762366" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2e39f68bd93a4493af7bfd27b8762366" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 46303 } } }
I20260620 11:46:36.876994 28545 sys_catalog.cc:458] T 00000000000000000000000000000000 P 2e39f68bd93a4493af7bfd27b8762366 [sys.catalog]: This master's current role is: LEADER
I20260620 11:46:36.876883 28544 sys_catalog.cc:455] T 00000000000000000000000000000000 P 2e39f68bd93a4493af7bfd27b8762366 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 2e39f68bd93a4493af7bfd27b8762366. Latest consensus state: current_term: 1 leader_uuid: "2e39f68bd93a4493af7bfd27b8762366" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2e39f68bd93a4493af7bfd27b8762366" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 46303 } } }
I20260620 11:46:36.877321 28544 sys_catalog.cc:458] T 00000000000000000000000000000000 P 2e39f68bd93a4493af7bfd27b8762366 [sys.catalog]: This master's current role is: LEADER
I20260620 11:46:36.877686 28551 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260620 11:46:36.878459 28551 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260620 11:46:36.879348 28543 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 11:46:36.877982 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.133:55051 (local address 127.0.0.1:46303)
0620 11:46:36.878017 (+ 35us) server_negotiation.cc:206] Beginning negotiation
0620 11:46:36.878019 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 11:46:36.878372 (+ 353us) server_negotiation.cc:407] Connection header received
0620 11:46:36.878421 (+ 49us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 11:46:36.878422 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 11:46:36.878427 (+ 5us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 11:46:36.878492 (+ 65us) server_negotiation.cc:226] Negotiated authn=SASL
0620 11:46:36.879135 (+ 643us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 11:46:36.879136 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 11:46:36.879137 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 11:46:36.879141 (+ 4us) server_negotiation.cc:922] calling sasl_server_start()
0620 11:46:36.879156 (+ 15us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 11:46:36.879161 (+ 5us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 11:46:36.879220 (+ 59us) server_negotiation.cc:1035] Waiting for connection context
0620 11:46:36.879283 (+ 63us) server_negotiation.cc:299] Negotiation successful
0620 11:46:36.879296 (+ 13us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":17}
I20260620 11:46:36.880043 28551 catalog_manager.cc:1347] Generated new cluster ID: e24e2a2380774a9e8b6192381e68f647
I20260620 11:46:36.880093 28551 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260620 11:46:36.918900 28551 catalog_manager.cc:1370] Generated new certificate authority record
I20260620 11:46:36.919095 28551 catalog_manager.cc:1504] Loading token signing keys...
I20260620 11:46:36.921155 28551 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 2e39f68bd93a4493af7bfd27b8762366: Generated new TSK 0
I20260620 11:46:36.921363 28551 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260620 11:46:36.927464 7083 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task1JRg7N/build/release/bin/kudu with pid 28471
2026-06-20T11:46:36Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/11 (1324 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/12
Loading random data
Initializing database '/tmp/dist-test-task1JRg7N/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1781955746769051-7083-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 20 11:46:36 dist-test-slave-cjc6 krb5kdc[28570](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 20 11:46:36 dist-test-slave-cjc6 krb5kdc[28570](info): set up 2 sockets
Jun 20 11:46:36 dist-test-slave-cjc6 krb5kdc[28570](info): commencing operation
krb5kdc: starting...
W20260620 11:46:37.961871 7083 mini_kdc.cc:121] Time spent starting KDC: real 1.017s user 0.000s sys 0.006s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task1JRg7N/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1781955746769051-7083-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task1JRg7N/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1781955746769051-7083-0/krb5kdc/test-user.keytab.
Jun 20 11:46:38 dist-test-slave-cjc6 krb5kdc[28570](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781955998, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-20T11:46:38Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-20T11:46:38Z Disabled control of system clock
WARNING: no policy specified for kudu/127.6.234.254@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.6.234.254@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.6.234.254 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task1JRg7N/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1781955746769051-7083-0/krb5kdc/kudu_127.6.234.254.keytab.
Entry for principal kudu/127.6.234.254 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task1JRg7N/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1781955746769051-7083-0/krb5kdc/kudu_127.6.234.254.keytab.
I20260620 11:46:38.080515 7083 external_mini_cluster.cc:1326] Running /tmp/dist-test-task1JRg7N/build/release/bin/kudu
/tmp/dist-test-task1JRg7N/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task1JRg7N/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1781955746769051-7083-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task1JRg7N/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1781955746769051-7083-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task1JRg7N/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1781955746769051-7083-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task1JRg7N/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1781955746769051-7083-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.6.234.254:34779
--webserver_interface=127.6.234.254
--webserver_port=0
--keytab_file=/tmp/dist-test-task1JRg7N/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1781955746769051-7083-0/krb5kdc/kudu_127.6.234.254.keytab
--principal=kudu/127.6.234.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.6.234.212:42889
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.6.234.254:34779
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=required
--rpc_encryption=required
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-task1JRg7N/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1781955746769051-7083-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task1JRg7N/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1781955746769051-7083-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task1JRg7N/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1781955746769051-7083-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 11:46:38.160529 28583 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 11:46:38.160763 28583 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 11:46:38.160830 28583 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 11:46:38.162048 28583 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 11:46:38.162089 28583 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 11:46:38.162104 28583 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 11:46:38.162117 28583 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 11:46:38.162129 28583 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 11:46:38.163599 28583 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.6.234.212:42889
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task1JRg7N/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1781955746769051-7083-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task1JRg7N/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1781955746769051-7083-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.6.234.254:34779
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task1JRg7N/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1781955746769051-7083-0/krb5kdc/kudu_127.6.234.254.keytab
--principal=kudu/127.6.234.254
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.6.234.254:34779
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task1JRg7N/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1781955746769051-7083-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.6.234.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.28583
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task1JRg7N/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1781955746769051-7083-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 20 Jun 2026 10:32:37 UTC on ccycloud.quasar-bgyofv.root.comops.site
build id 4457173
I20260620 11:46:38.163847 28583 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 11:46:38.164088 28583 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 11:46:38.168895 28591 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 11:46:38.168987 28588 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 11:46:38.168927 28589 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 11:46:38.169123 28583 server_base.cc:1034] running on GCE node
I20260620 11:46:38.169364 28583 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 11:46:38.169641 28583 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 11:46:38.170840 28583 hybrid_clock.cc:648] HybridClock initialized: now 1781955998170826 us; error 29 us; skew 500 ppm
Jun 20 11:46:38 dist-test-slave-cjc6 krb5kdc[28570](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781955998, etypes {rep=17 tkt=17 ses=17}, kudu/127.6.234.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 11:46:38.173323 28583 init.cc:377] Logged in from keytab as kudu/127.6.234.254@KRBTEST.COM (short username kudu)
I20260620 11:46:38.173893 28583 webserver.cc:460] Webserver started at http://127.6.234.254:35601/ using document root <none> and password file <none>
I20260620 11:46:38.174144 28583 fs_manager.cc:359] Metadata directory not provided
I20260620 11:46:38.174212 28583 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 11:46:38.174347 28583 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 11:46:38.175408 28583 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task1JRg7N/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1781955746769051-7083-0/minicluster-data/master-0/data/instance:
uuid: "361fea7c6a6a4b26bab94db035ee7108"
format_stamp: "Formatted at 2026-06-20 11:46:38 on dist-test-slave-cjc6"
server_key: "1357beb6eaec678be1688cc7044a4049"
server_key_iv: "ecc4cbc376a8f31900c0f96697a7ec67"
server_key_version: "encryptionkey@0"
I20260620 11:46:38.175776 28583 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task1JRg7N/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1781955746769051-7083-0/minicluster-data/master-0/wal/instance:
uuid: "361fea7c6a6a4b26bab94db035ee7108"
format_stamp: "Formatted at 2026-06-20 11:46:38 on dist-test-slave-cjc6"
server_key: "1357beb6eaec678be1688cc7044a4049"
server_key_iv: "ecc4cbc376a8f31900c0f96697a7ec67"
server_key_version: "encryptionkey@0"
I20260620 11:46:38.177102 28583 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260620 11:46:38.177887 28598 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 11:46:38.178074 28583 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260620 11:46:38.178166 28583 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task1JRg7N/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1781955746769051-7083-0/minicluster-data/master-0/data,/tmp/dist-test-task1JRg7N/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1781955746769051-7083-0/minicluster-data/master-0/wal
uuid: "361fea7c6a6a4b26bab94db035ee7108"
format_stamp: "Formatted at 2026-06-20 11:46:38 on dist-test-slave-cjc6"
server_key: "1357beb6eaec678be1688cc7044a4049"
server_key_iv: "ecc4cbc376a8f31900c0f96697a7ec67"
server_key_version: "encryptionkey@0"
I20260620 11:46:38.178248 28583 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task1JRg7N/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1781955746769051-7083-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task1JRg7N/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1781955746769051-7083-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task1JRg7N/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1781955746769051-7083-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 11:46:38.196012 28583 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 11:46:38.197978 28583 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 11:46:38.198139 28583 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 11:46:38.202109 28583 rpc_server.cc:307] RPC server started. Bound to: 127.6.234.254:34779
I20260620 11:46:38.202158 28650 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.6.234.254:34779 every 8 connection(s)
I20260620 11:46:38.202503 28583 server_base.cc:1166] Dumped server information to /tmp/dist-test-task1JRg7N/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1781955746769051-7083-0/minicluster-data/master-0/data/info.pb
I20260620 11:46:38.203328 28651 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 11:46:38.205085 7083 external_mini_cluster.cc:1388] Started /tmp/dist-test-task1JRg7N/build/release/bin/kudu as pid 28583
I20260620 11:46:38.205220 7083 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task1JRg7N/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1781955746769051-7083-0/minicluster-data/master-0/wal/instance
I20260620 11:46:38.205408 7083 external_mini_cluster.cc:1428] Setting key 397d949cc0c64da1cb42a6ed2e606a63
I20260620 11:46:38.205859 28651 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 361fea7c6a6a4b26bab94db035ee7108: Bootstrap starting.
I20260620 11:46:38.206516 28651 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 361fea7c6a6a4b26bab94db035ee7108: Neither blocks nor log segments found. Creating new log.
I20260620 11:46:38.206851 28651 log.cc:824] T 00000000000000000000000000000000 P 361fea7c6a6a4b26bab94db035ee7108: Log is configured to *not* fsync() on all Append() calls
I20260620 11:46:38.207607 28651 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 361fea7c6a6a4b26bab94db035ee7108: No bootstrap required, opened a new log
I20260620 11:46:38.209739 28651 raft_consensus.cc:348] T 00000000000000000000000000000000 P 361fea7c6a6a4b26bab94db035ee7108 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "361fea7c6a6a4b26bab94db035ee7108" member_type: VOTER last_known_addr { host: "127.6.234.254" port: 34779 } }
I20260620 11:46:38.209890 28651 raft_consensus.cc:374] T 00000000000000000000000000000000 P 361fea7c6a6a4b26bab94db035ee7108 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 11:46:38.209915 28651 raft_consensus.cc:729] T 00000000000000000000000000000000 P 361fea7c6a6a4b26bab94db035ee7108 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 361fea7c6a6a4b26bab94db035ee7108, State: Initialized, Role: FOLLOWER
I20260620 11:46:38.210062 28651 consensus_queue.cc:260] T 00000000000000000000000000000000 P 361fea7c6a6a4b26bab94db035ee7108 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "361fea7c6a6a4b26bab94db035ee7108" member_type: VOTER last_known_addr { host: "127.6.234.254" port: 34779 } }
I20260620 11:46:38.210131 28651 raft_consensus.cc:388] T 00000000000000000000000000000000 P 361fea7c6a6a4b26bab94db035ee7108 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260620 11:46:38.210152 28651 raft_consensus.cc:482] T 00000000000000000000000000000000 P 361fea7c6a6a4b26bab94db035ee7108 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260620 11:46:38.210217 28651 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 361fea7c6a6a4b26bab94db035ee7108 [term 0 FOLLOWER]: Advancing to term 1
I20260620 11:46:38.210958 28651 raft_consensus.cc:504] T 00000000000000000000000000000000 P 361fea7c6a6a4b26bab94db035ee7108 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "361fea7c6a6a4b26bab94db035ee7108" member_type: VOTER last_known_addr { host: "127.6.234.254" port: 34779 } }
I20260620 11:46:38.211073 28651 leader_election.cc:302] T 00000000000000000000000000000000 P 361fea7c6a6a4b26bab94db035ee7108 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 361fea7c6a6a4b26bab94db035ee7108; no voters:
I20260620 11:46:38.211233 28651 leader_election.cc:288] T 00000000000000000000000000000000 P 361fea7c6a6a4b26bab94db035ee7108 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260620 11:46:38.211263 28656 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 361fea7c6a6a4b26bab94db035ee7108 [term 1 FOLLOWER]: Leader election won for term 1
I20260620 11:46:38.211433 28656 raft_consensus.cc:686] T 00000000000000000000000000000000 P 361fea7c6a6a4b26bab94db035ee7108 [term 1 LEADER]: Becoming Leader. State: Replica: 361fea7c6a6a4b26bab94db035ee7108, State: Running, Role: LEADER
I20260620 11:46:38.211510 28651 sys_catalog.cc:564] T 00000000000000000000000000000000 P 361fea7c6a6a4b26bab94db035ee7108 [sys.catalog]: configured and running, proceeding with master startup.
I20260620 11:46:38.211552 28656 consensus_queue.cc:237] T 00000000000000000000000000000000 P 361fea7c6a6a4b26bab94db035ee7108 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "361fea7c6a6a4b26bab94db035ee7108" member_type: VOTER last_known_addr { host: "127.6.234.254" port: 34779 } }
I20260620 11:46:38.212016 28657 sys_catalog.cc:455] T 00000000000000000000000000000000 P 361fea7c6a6a4b26bab94db035ee7108 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "361fea7c6a6a4b26bab94db035ee7108" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "361fea7c6a6a4b26bab94db035ee7108" member_type: VOTER last_known_addr { host: "127.6.234.254" port: 34779 } } }
I20260620 11:46:38.212095 28657 sys_catalog.cc:458] T 00000000000000000000000000000000 P 361fea7c6a6a4b26bab94db035ee7108 [sys.catalog]: This master's current role is: LEADER
I20260620 11:46:38.212301 28658 sys_catalog.cc:455] T 00000000000000000000000000000000 P 361fea7c6a6a4b26bab94db035ee7108 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 361fea7c6a6a4b26bab94db035ee7108. Latest consensus state: current_term: 1 leader_uuid: "361fea7c6a6a4b26bab94db035ee7108" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "361fea7c6a6a4b26bab94db035ee7108" member_type: VOTER last_known_addr { host: "127.6.234.254" port: 34779 } } }
I20260620 11:46:38.212572 28658 sys_catalog.cc:458] T 00000000000000000000000000000000 P 361fea7c6a6a4b26bab94db035ee7108 [sys.catalog]: This master's current role is: LEADER
I20260620 11:46:38.212714 28663 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
Jun 20 11:46:38 dist-test-slave-cjc6 krb5kdc[28570](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781955998, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.6.234.254@KRBTEST.COM
I20260620 11:46:38.213172 28663 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260620 11:46:38.215520 28663 catalog_manager.cc:1347] Generated new cluster ID: c4e533f412c546138a0399cab5f2d155
I20260620 11:46:38.215586 28663 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260620 11:46:38.217041 28654 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 11:46:38.205951 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38723 (local address 127.6.234.254:34779)
0620 11:46:38.206127 (+ 176us) server_negotiation.cc:206] Beginning negotiation
0620 11:46:38.206143 (+ 16us) server_negotiation.cc:399] Waiting for connection header
0620 11:46:38.206171 (+ 28us) server_negotiation.cc:407] Connection header received
0620 11:46:38.206617 (+ 446us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 11:46:38.206625 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 11:46:38.206689 (+ 64us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 11:46:38.206805 (+ 116us) server_negotiation.cc:226] Negotiated authn=SASL
0620 11:46:38.207608 (+ 803us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 11:46:38.208527 (+ 919us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 11:46:38.209673 (+ 1146us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 11:46:38.210298 (+ 625us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 11:46:38.210388 (+ 90us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 11:46:38.213381 (+ 2993us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 11:46:38.213385 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 11:46:38.213386 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 11:46:38.213398 (+ 12us) server_negotiation.cc:922] calling sasl_server_start()
0620 11:46:38.215364 (+ 1966us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 11:46:38.215801 (+ 437us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 11:46:38.215804 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 11:46:38.215805 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 11:46:38.215904 (+ 99us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 11:46:38.216152 (+ 248us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 11:46:38.216153 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 11:46:38.216153 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 11:46:38.216432 (+ 279us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 11:46:38.216590 (+ 158us) server_negotiation.cc:1035] Waiting for connection context
0620 11:46:38.216758 (+ 168us) server_negotiation.cc:299] Negotiation successful
0620 11:46:38.216851 (+ 93us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":129,"thread_start_us":80,"threads_started":1}
I20260620 11:46:38.218025 7083 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260620 11:46:38.222622 28663 catalog_manager.cc:1370] Generated new certificate authority record
I20260620 11:46:38.223241 28663 catalog_manager.cc:1504] Loading token signing keys...
I20260620 11:46:38.225839 28663 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 361fea7c6a6a4b26bab94db035ee7108: Generated new TSK 0
I20260620 11:46:38.226009 28663 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260620 11:46:38.226622 28654 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 11:46:38.219571 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54657 (local address 127.6.234.254:34779)
0620 11:46:38.219613 (+ 42us) server_negotiation.cc:206] Beginning negotiation
0620 11:46:38.219616 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0620 11:46:38.219625 (+ 9us) server_negotiation.cc:407] Connection header received
0620 11:46:38.219644 (+ 19us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 11:46:38.219645 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 11:46:38.219653 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 11:46:38.219713 (+ 60us) server_negotiation.cc:226] Negotiated authn=SASL
0620 11:46:38.220291 (+ 578us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 11:46:38.221049 (+ 758us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 11:46:38.222046 (+ 997us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 11:46:38.222352 (+ 306us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 11:46:38.222422 (+ 70us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 11:46:38.223826 (+ 1404us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 11:46:38.223828 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 11:46:38.223828 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 11:46:38.223835 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0620 11:46:38.225365 (+ 1530us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 11:46:38.225812 (+ 447us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 11:46:38.225813 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 11:46:38.225814 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 11:46:38.225867 (+ 53us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 11:46:38.226108 (+ 241us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 11:46:38.226108 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 11:46:38.226109 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 11:46:38.226220 (+ 111us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 11:46:38.226295 (+ 75us) server_negotiation.cc:1035] Waiting for connection context
0620 11:46:38.226519 (+ 224us) server_negotiation.cc:299] Negotiation successful
0620 11:46:38.226571 (+ 52us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":23}
I20260620 11:46:38.228420 7083 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task1JRg7N/build/release/bin/kudu with pid 28583
W20260620 11:46:38.232141 28568 connection.cc:535] client connection to 127.6.234.254:34779 recv error: Network error: failed to read from TLS socket (remote: 127.6.234.254:34779): Connection reset by peer (error 104)
2026-06-20T11:46:38Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/12 (1300 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/13
Loading random data
Initializing database '/tmp/dist-test-task1JRg7N/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1781955746769051-7083-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 20 11:46:38 dist-test-slave-cjc6 krb5kdc[28682](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 20 11:46:38 dist-test-slave-cjc6 krb5kdc[28682](info): set up 2 sockets
Jun 20 11:46:38 dist-test-slave-cjc6 krb5kdc[28682](info): commencing operation
krb5kdc: starting...
W20260620 11:46:39.266024 7083 mini_kdc.cc:121] Time spent starting KDC: real 1.020s user 0.000s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task1JRg7N/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1781955746769051-7083-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task1JRg7N/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1781955746769051-7083-0/krb5kdc/test-user.keytab.
Jun 20 11:46:39 dist-test-slave-cjc6 krb5kdc[28682](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781955999, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-20T11:46:39Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-20T11:46:39Z Disabled control of system clock
WARNING: no policy specified for kudu/127.6.234.254@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.6.234.254@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.6.234.254 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task1JRg7N/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1781955746769051-7083-0/krb5kdc/kudu_127.6.234.254.keytab.
Entry for principal kudu/127.6.234.254 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task1JRg7N/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1781955746769051-7083-0/krb5kdc/kudu_127.6.234.254.keytab.
I20260620 11:46:39.382563 7083 external_mini_cluster.cc:1326] Running /tmp/dist-test-task1JRg7N/build/release/bin/kudu
/tmp/dist-test-task1JRg7N/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task1JRg7N/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1781955746769051-7083-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task1JRg7N/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1781955746769051-7083-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task1JRg7N/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1781955746769051-7083-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task1JRg7N/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1781955746769051-7083-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.6.234.254:41045
--webserver_interface=127.6.234.254
--webserver_port=0
--keytab_file=/tmp/dist-test-task1JRg7N/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1781955746769051-7083-0/krb5kdc/kudu_127.6.234.254.keytab
--principal=kudu/127.6.234.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.6.234.212:32967
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.6.234.254:41045
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=required
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-task1JRg7N/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1781955746769051-7083-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task1JRg7N/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1781955746769051-7083-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task1JRg7N/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1781955746769051-7083-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 11:46:39.462502 28695 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 11:46:39.462740 28695 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 11:46:39.462766 28695 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 11:46:39.463953 28695 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 11:46:39.463997 28695 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 11:46:39.464011 28695 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 11:46:39.464025 28695 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 11:46:39.464036 28695 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 11:46:39.465442 28695 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.6.234.212:32967
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task1JRg7N/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1781955746769051-7083-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task1JRg7N/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1781955746769051-7083-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.6.234.254:41045
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task1JRg7N/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1781955746769051-7083-0/krb5kdc/kudu_127.6.234.254.keytab
--principal=kudu/127.6.234.254
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.6.234.254:41045
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task1JRg7N/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1781955746769051-7083-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.6.234.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.28695
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task1JRg7N/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1781955746769051-7083-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 20 Jun 2026 10:32:37 UTC on ccycloud.quasar-bgyofv.root.comops.site
build id 4457173
I20260620 11:46:39.465636 28695 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 11:46:39.465843 28695 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 11:46:39.470448 28701 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 11:46:39.470443 28703 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 11:46:39.470701 28695 server_base.cc:1034] running on GCE node
W20260620 11:46:39.470788 28700 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 11:46:39.471067 28695 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 11:46:39.471355 28695 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 11:46:39.472525 28695 hybrid_clock.cc:648] HybridClock initialized: now 1781955999472519 us; error 36 us; skew 500 ppm
Jun 20 11:46:39 dist-test-slave-cjc6 krb5kdc[28682](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781955999, etypes {rep=17 tkt=17 ses=17}, kudu/127.6.234.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 11:46:39.475106 28695 init.cc:377] Logged in from keytab as kudu/127.6.234.254@KRBTEST.COM (short username kudu)
I20260620 11:46:39.475602 28695 webserver.cc:460] Webserver started at http://127.6.234.254:34153/ using document root <none> and password file <none>
I20260620 11:46:39.475847 28695 fs_manager.cc:359] Metadata directory not provided
I20260620 11:46:39.475916 28695 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 11:46:39.476068 28695 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 11:46:39.477150 28695 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task1JRg7N/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1781955746769051-7083-0/minicluster-data/master-0/data/instance:
uuid: "00e39cb7ebc94376924ca9e588823b63"
format_stamp: "Formatted at 2026-06-20 11:46:39 on dist-test-slave-cjc6"
server_key: "1e1ffa0373c2d912d9c2552eb2fc107c"
server_key_iv: "483ca79c14d504b50309a2ad0473b61b"
server_key_version: "encryptionkey@0"
I20260620 11:46:39.477519 28695 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task1JRg7N/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1781955746769051-7083-0/minicluster-data/master-0/wal/instance:
uuid: "00e39cb7ebc94376924ca9e588823b63"
format_stamp: "Formatted at 2026-06-20 11:46:39 on dist-test-slave-cjc6"
server_key: "1e1ffa0373c2d912d9c2552eb2fc107c"
server_key_iv: "483ca79c14d504b50309a2ad0473b61b"
server_key_version: "encryptionkey@0"
I20260620 11:46:39.478892 28695 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.001s
I20260620 11:46:39.479637 28710 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 11:46:39.479828 28695 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.001s
I20260620 11:46:39.479925 28695 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task1JRg7N/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1781955746769051-7083-0/minicluster-data/master-0/data,/tmp/dist-test-task1JRg7N/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1781955746769051-7083-0/minicluster-data/master-0/wal
uuid: "00e39cb7ebc94376924ca9e588823b63"
format_stamp: "Formatted at 2026-06-20 11:46:39 on dist-test-slave-cjc6"
server_key: "1e1ffa0373c2d912d9c2552eb2fc107c"
server_key_iv: "483ca79c14d504b50309a2ad0473b61b"
server_key_version: "encryptionkey@0"
I20260620 11:46:39.480016 28695 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task1JRg7N/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1781955746769051-7083-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task1JRg7N/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1781955746769051-7083-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task1JRg7N/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1781955746769051-7083-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 11:46:39.491894 28695 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 11:46:39.494223 28695 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 11:46:39.494391 28695 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 11:46:39.498468 28695 rpc_server.cc:307] RPC server started. Bound to: 127.6.234.254:41045
I20260620 11:46:39.498504 28762 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.6.234.254:41045 every 8 connection(s)
I20260620 11:46:39.498879 28695 server_base.cc:1166] Dumped server information to /tmp/dist-test-task1JRg7N/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1781955746769051-7083-0/minicluster-data/master-0/data/info.pb
I20260620 11:46:39.499758 28763 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 11:46:39.502125 28763 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 00e39cb7ebc94376924ca9e588823b63: Bootstrap starting.
I20260620 11:46:39.502727 28763 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 00e39cb7ebc94376924ca9e588823b63: Neither blocks nor log segments found. Creating new log.
I20260620 11:46:39.503008 28763 log.cc:824] T 00000000000000000000000000000000 P 00e39cb7ebc94376924ca9e588823b63: Log is configured to *not* fsync() on all Append() calls
I20260620 11:46:39.503815 28763 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 00e39cb7ebc94376924ca9e588823b63: No bootstrap required, opened a new log
I20260620 11:46:39.505566 28763 raft_consensus.cc:348] T 00000000000000000000000000000000 P 00e39cb7ebc94376924ca9e588823b63 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "00e39cb7ebc94376924ca9e588823b63" member_type: VOTER last_known_addr { host: "127.6.234.254" port: 41045 } }
I20260620 11:46:39.505777 28763 raft_consensus.cc:374] T 00000000000000000000000000000000 P 00e39cb7ebc94376924ca9e588823b63 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 11:46:39.505808 28763 raft_consensus.cc:729] T 00000000000000000000000000000000 P 00e39cb7ebc94376924ca9e588823b63 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 00e39cb7ebc94376924ca9e588823b63, State: Initialized, Role: FOLLOWER
I20260620 11:46:39.505980 28763 consensus_queue.cc:260] T 00000000000000000000000000000000 P 00e39cb7ebc94376924ca9e588823b63 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "00e39cb7ebc94376924ca9e588823b63" member_type: VOTER last_known_addr { host: "127.6.234.254" port: 41045 } }
I20260620 11:46:39.506047 28763 raft_consensus.cc:388] T 00000000000000000000000000000000 P 00e39cb7ebc94376924ca9e588823b63 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260620 11:46:39.506104 28763 raft_consensus.cc:482] T 00000000000000000000000000000000 P 00e39cb7ebc94376924ca9e588823b63 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260620 11:46:39.506175 28763 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 00e39cb7ebc94376924ca9e588823b63 [term 0 FOLLOWER]: Advancing to term 1
I20260620 11:46:39.507138 28763 raft_consensus.cc:504] T 00000000000000000000000000000000 P 00e39cb7ebc94376924ca9e588823b63 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "00e39cb7ebc94376924ca9e588823b63" member_type: VOTER last_known_addr { host: "127.6.234.254" port: 41045 } }
I20260620 11:46:39.507266 7083 external_mini_cluster.cc:1388] Started /tmp/dist-test-task1JRg7N/build/release/bin/kudu as pid 28695
I20260620 11:46:39.507273 28763 leader_election.cc:302] T 00000000000000000000000000000000 P 00e39cb7ebc94376924ca9e588823b63 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 00e39cb7ebc94376924ca9e588823b63; no voters:
I20260620 11:46:39.507405 7083 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task1JRg7N/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1781955746769051-7083-0/minicluster-data/master-0/wal/instance
I20260620 11:46:39.507457 28763 leader_election.cc:288] T 00000000000000000000000000000000 P 00e39cb7ebc94376924ca9e588823b63 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260620 11:46:39.507524 28766 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 00e39cb7ebc94376924ca9e588823b63 [term 1 FOLLOWER]: Leader election won for term 1
I20260620 11:46:39.507660 7083 external_mini_cluster.cc:1428] Setting key 3435d02959e8f338f3e87f0498d63a56
I20260620 11:46:39.507719 28766 raft_consensus.cc:686] T 00000000000000000000000000000000 P 00e39cb7ebc94376924ca9e588823b63 [term 1 LEADER]: Becoming Leader. State: Replica: 00e39cb7ebc94376924ca9e588823b63, State: Running, Role: LEADER
I20260620 11:46:39.507735 28763 sys_catalog.cc:564] T 00000000000000000000000000000000 P 00e39cb7ebc94376924ca9e588823b63 [sys.catalog]: configured and running, proceeding with master startup.
I20260620 11:46:39.507808 28766 consensus_queue.cc:237] T 00000000000000000000000000000000 P 00e39cb7ebc94376924ca9e588823b63 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "00e39cb7ebc94376924ca9e588823b63" member_type: VOTER last_known_addr { host: "127.6.234.254" port: 41045 } }
I20260620 11:46:39.508117 28767 sys_catalog.cc:455] T 00000000000000000000000000000000 P 00e39cb7ebc94376924ca9e588823b63 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "00e39cb7ebc94376924ca9e588823b63" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "00e39cb7ebc94376924ca9e588823b63" member_type: VOTER last_known_addr { host: "127.6.234.254" port: 41045 } } }
I20260620 11:46:39.508109 28766 sys_catalog.cc:455] T 00000000000000000000000000000000 P 00e39cb7ebc94376924ca9e588823b63 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 00e39cb7ebc94376924ca9e588823b63. Latest consensus state: current_term: 1 leader_uuid: "00e39cb7ebc94376924ca9e588823b63" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "00e39cb7ebc94376924ca9e588823b63" member_type: VOTER last_known_addr { host: "127.6.234.254" port: 41045 } } }
I20260620 11:46:39.508210 28767 sys_catalog.cc:458] T 00000000000000000000000000000000 P 00e39cb7ebc94376924ca9e588823b63 [sys.catalog]: This master's current role is: LEADER
I20260620 11:46:39.508215 28766 sys_catalog.cc:458] T 00000000000000000000000000000000 P 00e39cb7ebc94376924ca9e588823b63 [sys.catalog]: This master's current role is: LEADER
I20260620 11:46:39.508412 28773 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260620 11:46:39.508985 28773 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260620 11:46:39.510661 28773 catalog_manager.cc:1347] Generated new cluster ID: 1bf010d2bcec475ab9bfeece649128a3
I20260620 11:46:39.510720 28773 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260620 11:46:39.514755 28777 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 11:46:39.508532 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40909 (local address 127.6.234.254:41045)
0620 11:46:39.508745 (+ 213us) server_negotiation.cc:206] Beginning negotiation
0620 11:46:39.508754 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0620 11:46:39.509080 (+ 326us) server_negotiation.cc:407] Connection header received
0620 11:46:39.509224 (+ 144us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 11:46:39.509232 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 11:46:39.509315 (+ 83us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 11:46:39.509480 (+ 165us) server_negotiation.cc:226] Negotiated authn=SASL
0620 11:46:39.509924 (+ 444us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 11:46:39.511255 (+ 1331us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 11:46:39.512452 (+ 1197us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 11:46:39.513089 (+ 637us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 11:46:39.513219 (+ 130us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 11:46:39.514011 (+ 792us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 11:46:39.514016 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 11:46:39.514016 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 11:46:39.514241 (+ 225us) server_negotiation.cc:922] calling sasl_server_start()
0620 11:46:39.514281 (+ 40us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 11:46:39.514296 (+ 15us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 11:46:39.514383 (+ 87us) server_negotiation.cc:1035] Waiting for connection context
0620 11:46:39.514488 (+ 105us) server_negotiation.cc:299] Negotiation successful
0620 11:46:39.514520 (+ 32us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":145,"thread_start_us":73,"threads_started":1}
I20260620 11:46:39.515812 7083 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260620 11:46:39.521108 28777 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 11:46:39.517417 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44705 (local address 127.6.234.254:41045)
0620 11:46:39.517456 (+ 39us) server_negotiation.cc:206] Beginning negotiation
0620 11:46:39.517458 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 11:46:39.517623 (+ 165us) server_negotiation.cc:407] Connection header received
0620 11:46:39.517672 (+ 49us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 11:46:39.517673 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 11:46:39.517684 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 11:46:39.517779 (+ 95us) server_negotiation.cc:226] Negotiated authn=SASL
0620 11:46:39.517934 (+ 155us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 11:46:39.518723 (+ 789us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 11:46:39.519718 (+ 995us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 11:46:39.520042 (+ 324us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 11:46:39.520104 (+ 62us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 11:46:39.520855 (+ 751us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 11:46:39.520856 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 11:46:39.520856 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 11:46:39.520861 (+ 5us) server_negotiation.cc:922] calling sasl_server_start()
0620 11:46:39.520875 (+ 14us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 11:46:39.520880 (+ 5us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 11:46:39.520940 (+ 60us) server_negotiation.cc:1035] Waiting for connection context
0620 11:46:39.521019 (+ 79us) server_negotiation.cc:299] Negotiation successful
0620 11:46:39.521031 (+ 12us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":21}
I20260620 11:46:39.523698 28773 catalog_manager.cc:1370] Generated new certificate authority record
I20260620 11:46:39.524236 28773 catalog_manager.cc:1504] Loading token signing keys...
I20260620 11:46:39.534340 28773 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 00e39cb7ebc94376924ca9e588823b63: Generated new TSK 0
I20260620 11:46:39.534575 28773 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260620 11:46:39.541200 7083 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task1JRg7N/build/release/bin/kudu with pid 28695
W20260620 11:46:39.545588 28680 connection.cc:535] client connection to 127.6.234.254:41045 recv error: Network error: failed to read from TLS socket (remote: 127.6.234.254:41045): Connection reset by peer (error 104)
2026-06-20T11:46:39Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/13 (1313 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/14
Loading random data
Initializing database '/tmp/dist-test-task1JRg7N/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1781955746769051-7083-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 20 11:46:39 dist-test-slave-cjc6 krb5kdc[28794](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 20 11:46:39 dist-test-slave-cjc6 krb5kdc[28794](info): set up 2 sockets
Jun 20 11:46:39 dist-test-slave-cjc6 krb5kdc[28794](info): commencing operation
krb5kdc: starting...
W20260620 11:46:40.581684 7083 mini_kdc.cc:121] Time spent starting KDC: real 1.022s user 0.000s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task1JRg7N/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1781955746769051-7083-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task1JRg7N/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1781955746769051-7083-0/krb5kdc/test-user.keytab.
Jun 20 11:46:40 dist-test-slave-cjc6 krb5kdc[28794](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781956000, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-20T11:46:40Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-20T11:46:40Z Disabled control of system clock
WARNING: no policy specified for kudu/127.6.234.254@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.6.234.254@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.6.234.254 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task1JRg7N/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1781955746769051-7083-0/krb5kdc/kudu_127.6.234.254.keytab.
Entry for principal kudu/127.6.234.254 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task1JRg7N/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1781955746769051-7083-0/krb5kdc/kudu_127.6.234.254.keytab.
I20260620 11:46:40.698758 7083 external_mini_cluster.cc:1326] Running /tmp/dist-test-task1JRg7N/build/release/bin/kudu
/tmp/dist-test-task1JRg7N/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task1JRg7N/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1781955746769051-7083-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task1JRg7N/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1781955746769051-7083-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task1JRg7N/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1781955746769051-7083-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task1JRg7N/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1781955746769051-7083-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.6.234.254:41309
--webserver_interface=127.6.234.254
--webserver_port=0
--keytab_file=/tmp/dist-test-task1JRg7N/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1781955746769051-7083-0/krb5kdc/kudu_127.6.234.254.keytab
--principal=kudu/127.6.234.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.6.234.212:35087
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.6.234.254:41309
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=disabled
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-task1JRg7N/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1781955746769051-7083-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task1JRg7N/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1781955746769051-7083-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task1JRg7N/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1781955746769051-7083-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 11:46:40.780561 28807 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 11:46:40.780788 28807 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 11:46:40.780812 28807 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 11:46:40.781989 28807 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 11:46:40.782032 28807 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 11:46:40.782047 28807 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 11:46:40.782060 28807 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 11:46:40.782073 28807 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 11:46:40.783591 28807 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.6.234.212:35087
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task1JRg7N/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1781955746769051-7083-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task1JRg7N/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1781955746769051-7083-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.6.234.254:41309
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task1JRg7N/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1781955746769051-7083-0/krb5kdc/kudu_127.6.234.254.keytab
--principal=kudu/127.6.234.254
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.6.234.254:41309
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task1JRg7N/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1781955746769051-7083-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.6.234.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.28807
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task1JRg7N/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1781955746769051-7083-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 20 Jun 2026 10:32:37 UTC on ccycloud.quasar-bgyofv.root.comops.site
build id 4457173
I20260620 11:46:40.783798 28807 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 11:46:40.783964 28807 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 11:46:40.788678 28813 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 11:46:40.788700 28815 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 11:46:40.788713 28812 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 11:46:40.788748 28807 server_base.cc:1034] running on GCE node
I20260620 11:46:40.789126 28807 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 11:46:40.789422 28807 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 11:46:40.790565 28807 hybrid_clock.cc:648] HybridClock initialized: now 1781956000790529 us; error 58 us; skew 500 ppm
Jun 20 11:46:40 dist-test-slave-cjc6 krb5kdc[28794](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781956000, etypes {rep=17 tkt=17 ses=17}, kudu/127.6.234.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 11:46:40.793036 28807 init.cc:377] Logged in from keytab as kudu/127.6.234.254@KRBTEST.COM (short username kudu)
I20260620 11:46:40.793463 28807 webserver.cc:460] Webserver started at http://127.6.234.254:36463/ using document root <none> and password file <none>
I20260620 11:46:40.793766 28807 fs_manager.cc:359] Metadata directory not provided
I20260620 11:46:40.793838 28807 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 11:46:40.793993 28807 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 11:46:40.795043 28807 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task1JRg7N/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1781955746769051-7083-0/minicluster-data/master-0/data/instance:
uuid: "198bf8402df74a2090148abb5956803a"
format_stamp: "Formatted at 2026-06-20 11:46:40 on dist-test-slave-cjc6"
server_key: "22dd22b74ba27bab72cfce785bd1dc15"
server_key_iv: "d9c41be84491d57df166840c2874f666"
server_key_version: "encryptionkey@0"
I20260620 11:46:40.795431 28807 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task1JRg7N/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1781955746769051-7083-0/minicluster-data/master-0/wal/instance:
uuid: "198bf8402df74a2090148abb5956803a"
format_stamp: "Formatted at 2026-06-20 11:46:40 on dist-test-slave-cjc6"
server_key: "22dd22b74ba27bab72cfce785bd1dc15"
server_key_iv: "d9c41be84491d57df166840c2874f666"
server_key_version: "encryptionkey@0"
I20260620 11:46:40.796754 28807 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.003s
I20260620 11:46:40.797492 28822 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 11:46:40.797659 28807 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.001s
I20260620 11:46:40.797787 28807 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task1JRg7N/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1781955746769051-7083-0/minicluster-data/master-0/data,/tmp/dist-test-task1JRg7N/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1781955746769051-7083-0/minicluster-data/master-0/wal
uuid: "198bf8402df74a2090148abb5956803a"
format_stamp: "Formatted at 2026-06-20 11:46:40 on dist-test-slave-cjc6"
server_key: "22dd22b74ba27bab72cfce785bd1dc15"
server_key_iv: "d9c41be84491d57df166840c2874f666"
server_key_version: "encryptionkey@0"
I20260620 11:46:40.797863 28807 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task1JRg7N/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1781955746769051-7083-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task1JRg7N/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1781955746769051-7083-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task1JRg7N/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1781955746769051-7083-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 11:46:40.801126 28807 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 11:46:40.803012 28807 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 11:46:40.803175 28807 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 11:46:40.807026 28807 rpc_server.cc:307] RPC server started. Bound to: 127.6.234.254:41309
I20260620 11:46:40.807066 28874 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.6.234.254:41309 every 8 connection(s)
I20260620 11:46:40.807399 28807 server_base.cc:1166] Dumped server information to /tmp/dist-test-task1JRg7N/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1781955746769051-7083-0/minicluster-data/master-0/data/info.pb
I20260620 11:46:40.808497 28875 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 11:46:40.810737 28875 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 198bf8402df74a2090148abb5956803a: Bootstrap starting.
I20260620 11:46:40.811244 28875 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 198bf8402df74a2090148abb5956803a: Neither blocks nor log segments found. Creating new log.
I20260620 11:46:40.811477 28875 log.cc:824] T 00000000000000000000000000000000 P 198bf8402df74a2090148abb5956803a: Log is configured to *not* fsync() on all Append() calls
I20260620 11:46:40.812129 28875 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 198bf8402df74a2090148abb5956803a: No bootstrap required, opened a new log
I20260620 11:46:40.813442 7083 external_mini_cluster.cc:1388] Started /tmp/dist-test-task1JRg7N/build/release/bin/kudu as pid 28807
I20260620 11:46:40.813555 7083 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task1JRg7N/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1781955746769051-7083-0/minicluster-data/master-0/wal/instance
I20260620 11:46:40.813776 7083 external_mini_cluster.cc:1428] Setting key 08f7089d6188518158e5e45271fbf63f
I20260620 11:46:40.813984 28875 raft_consensus.cc:348] T 00000000000000000000000000000000 P 198bf8402df74a2090148abb5956803a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "198bf8402df74a2090148abb5956803a" member_type: VOTER last_known_addr { host: "127.6.234.254" port: 41309 } }
I20260620 11:46:40.814146 28875 raft_consensus.cc:374] T 00000000000000000000000000000000 P 198bf8402df74a2090148abb5956803a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 11:46:40.814179 28875 raft_consensus.cc:729] T 00000000000000000000000000000000 P 198bf8402df74a2090148abb5956803a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 198bf8402df74a2090148abb5956803a, State: Initialized, Role: FOLLOWER
I20260620 11:46:40.814393 28875 consensus_queue.cc:260] T 00000000000000000000000000000000 P 198bf8402df74a2090148abb5956803a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "198bf8402df74a2090148abb5956803a" member_type: VOTER last_known_addr { host: "127.6.234.254" port: 41309 } }
I20260620 11:46:40.814491 28875 raft_consensus.cc:388] T 00000000000000000000000000000000 P 198bf8402df74a2090148abb5956803a [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260620 11:46:40.814527 28875 raft_consensus.cc:482] T 00000000000000000000000000000000 P 198bf8402df74a2090148abb5956803a [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260620 11:46:40.814576 28875 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 198bf8402df74a2090148abb5956803a [term 0 FOLLOWER]: Advancing to term 1
I20260620 11:46:40.815662 28875 raft_consensus.cc:504] T 00000000000000000000000000000000 P 198bf8402df74a2090148abb5956803a [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "198bf8402df74a2090148abb5956803a" member_type: VOTER last_known_addr { host: "127.6.234.254" port: 41309 } }
I20260620 11:46:40.815796 28875 leader_election.cc:302] T 00000000000000000000000000000000 P 198bf8402df74a2090148abb5956803a [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 198bf8402df74a2090148abb5956803a; no voters:
I20260620 11:46:40.815943 28879 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 11:46:40.814301 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53497 (local address 127.6.234.254:41309)
0620 11:46:40.814482 (+ 181us) server_negotiation.cc:206] Beginning negotiation
0620 11:46:40.814492 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0620 11:46:40.814530 (+ 38us) server_negotiation.cc:407] Connection header received
0620 11:46:40.814659 (+ 129us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 11:46:40.814666 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 11:46:40.814698 (+ 32us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 11:46:40.814862 (+ 164us) server_negotiation.cc:226] Negotiated authn=SASL
0620 11:46:40.815507 (+ 645us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 11:46:40.815511 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 11:46:40.815511 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 11:46:40.815516 (+ 5us) server_negotiation.cc:922] calling sasl_server_start()
0620 11:46:40.815561 (+ 45us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 11:46:40.815576 (+ 15us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 11:46:40.815652 (+ 76us) server_negotiation.cc:1035] Waiting for connection context
0620 11:46:40.815705 (+ 53us) server_negotiation.cc:299] Negotiation successful
0620 11:46:40.815746 (+ 41us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":134,"thread_start_us":80,"threads_started":1}
I20260620 11:46:40.815958 28875 leader_election.cc:288] T 00000000000000000000000000000000 P 198bf8402df74a2090148abb5956803a [CANDIDATE]: Term 1 election: Requested vote from peers
I20260620 11:46:40.816043 28880 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 198bf8402df74a2090148abb5956803a [term 1 FOLLOWER]: Leader election won for term 1
I20260620 11:46:40.816185 28875 sys_catalog.cc:564] T 00000000000000000000000000000000 P 198bf8402df74a2090148abb5956803a [sys.catalog]: configured and running, proceeding with master startup.
I20260620 11:46:40.816219 28880 raft_consensus.cc:686] T 00000000000000000000000000000000 P 198bf8402df74a2090148abb5956803a [term 1 LEADER]: Becoming Leader. State: Replica: 198bf8402df74a2090148abb5956803a, State: Running, Role: LEADER
I20260620 11:46:40.816354 28880 consensus_queue.cc:237] T 00000000000000000000000000000000 P 198bf8402df74a2090148abb5956803a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "198bf8402df74a2090148abb5956803a" member_type: VOTER last_known_addr { host: "127.6.234.254" port: 41309 } }
I20260620 11:46:40.816951 7083 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260620 11:46:40.818004 28881 sys_catalog.cc:455] T 00000000000000000000000000000000 P 198bf8402df74a2090148abb5956803a [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "198bf8402df74a2090148abb5956803a" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "198bf8402df74a2090148abb5956803a" member_type: VOTER last_known_addr { host: "127.6.234.254" port: 41309 } } }
I20260620 11:46:40.818223 28882 sys_catalog.cc:455] T 00000000000000000000000000000000 P 198bf8402df74a2090148abb5956803a [sys.catalog]: SysCatalogTable state changed. Reason: New leader 198bf8402df74a2090148abb5956803a. Latest consensus state: current_term: 1 leader_uuid: "198bf8402df74a2090148abb5956803a" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "198bf8402df74a2090148abb5956803a" member_type: VOTER last_known_addr { host: "127.6.234.254" port: 41309 } } }
I20260620 11:46:40.818639 28882 sys_catalog.cc:458] T 00000000000000000000000000000000 P 198bf8402df74a2090148abb5956803a [sys.catalog]: This master's current role is: LEADER
I20260620 11:46:40.818315 28881 sys_catalog.cc:458] T 00000000000000000000000000000000 P 198bf8402df74a2090148abb5956803a [sys.catalog]: This master's current role is: LEADER
W20260620 11:46:40.818547 28895 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 198bf8402df74a2090148abb5956803a: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260620 11:46:40.818849 28895 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260620 11:46:40.819020 28902 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260620 11:46:40.819181 28902 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260620 11:46:40.819797 28879 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 11:46:40.818502 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49191 (local address 127.6.234.254:41309)
0620 11:46:40.818538 (+ 36us) server_negotiation.cc:206] Beginning negotiation
0620 11:46:40.818539 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0620 11:46:40.818740 (+ 201us) server_negotiation.cc:407] Connection header received
0620 11:46:40.818845 (+ 105us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 11:46:40.818846 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 11:46:40.818852 (+ 6us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 11:46:40.818881 (+ 29us) server_negotiation.cc:226] Negotiated authn=SASL
0620 11:46:40.819586 (+ 705us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 11:46:40.819588 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 11:46:40.819588 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 11:46:40.819592 (+ 4us) server_negotiation.cc:922] calling sasl_server_start()
0620 11:46:40.819606 (+ 14us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 11:46:40.819610 (+ 4us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 11:46:40.819662 (+ 52us) server_negotiation.cc:1035] Waiting for connection context
0620 11:46:40.819714 (+ 52us) server_negotiation.cc:299] Negotiation successful
0620 11:46:40.819728 (+ 14us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":17}
I20260620 11:46:40.820714 28902 catalog_manager.cc:1347] Generated new cluster ID: a212250e78f443a1a783c3d1c1b9c592
I20260620 11:46:40.820788 28902 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260620 11:46:40.835013 28902 catalog_manager.cc:1370] Generated new certificate authority record
I20260620 11:46:40.835215 28902 catalog_manager.cc:1504] Loading token signing keys...
I20260620 11:46:40.838804 28902 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 198bf8402df74a2090148abb5956803a: Generated new TSK 0
I20260620 11:46:40.838984 28902 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260620 11:46:40.845240 7083 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task1JRg7N/build/release/bin/kudu with pid 28807
2026-06-20T11:46:40Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/14 (1303 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/15
Loading random data
Initializing database '/tmp/dist-test-task1JRg7N/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1781955746769051-7083-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 20 11:46:40 dist-test-slave-cjc6 krb5kdc[28906](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 20 11:46:40 dist-test-slave-cjc6 krb5kdc[28906](info): set up 2 sockets
Jun 20 11:46:40 dist-test-slave-cjc6 krb5kdc[28906](info): commencing operation
krb5kdc: starting...
W20260620 11:46:41.885969 7083 mini_kdc.cc:121] Time spent starting KDC: real 1.023s user 0.000s sys 0.006s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task1JRg7N/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1781955746769051-7083-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task1JRg7N/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1781955746769051-7083-0/krb5kdc/test-user.keytab.
Jun 20 11:46:41 dist-test-slave-cjc6 krb5kdc[28906](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781956001, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-20T11:46:41Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-20T11:46:41Z Disabled control of system clock
WARNING: no policy specified for kudu/127.6.234.254@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.6.234.254@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.6.234.254 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task1JRg7N/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1781955746769051-7083-0/krb5kdc/kudu_127.6.234.254.keytab.
Entry for principal kudu/127.6.234.254 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task1JRg7N/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1781955746769051-7083-0/krb5kdc/kudu_127.6.234.254.keytab.
I20260620 11:46:42.000895 7083 external_mini_cluster.cc:1326] Running /tmp/dist-test-task1JRg7N/build/release/bin/kudu
/tmp/dist-test-task1JRg7N/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task1JRg7N/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1781955746769051-7083-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task1JRg7N/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1781955746769051-7083-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task1JRg7N/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1781955746769051-7083-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task1JRg7N/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1781955746769051-7083-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.6.234.254:37845
--webserver_interface=127.6.234.254
--webserver_port=0
--keytab_file=/tmp/dist-test-task1JRg7N/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1781955746769051-7083-0/krb5kdc/kudu_127.6.234.254.keytab
--principal=kudu/127.6.234.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.6.234.212:44391
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.6.234.254:37845
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=required
--rpc_encryption=required
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-task1JRg7N/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1781955746769051-7083-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task1JRg7N/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1781955746769051-7083-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task1JRg7N/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1781955746769051-7083-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 11:46:42.082809 28919 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 11:46:42.083070 28919 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 11:46:42.083109 28919 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 11:46:42.084362 28919 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 11:46:42.084438 28919 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 11:46:42.084465 28919 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 11:46:42.084496 28919 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 11:46:42.084522 28919 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 11:46:42.086097 28919 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.6.234.212:44391
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task1JRg7N/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1781955746769051-7083-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task1JRg7N/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1781955746769051-7083-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.6.234.254:37845
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task1JRg7N/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1781955746769051-7083-0/krb5kdc/kudu_127.6.234.254.keytab
--principal=kudu/127.6.234.254
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.6.234.254:37845
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task1JRg7N/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1781955746769051-7083-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.6.234.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.28919
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task1JRg7N/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1781955746769051-7083-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 20 Jun 2026 10:32:37 UTC on ccycloud.quasar-bgyofv.root.comops.site
build id 4457173
I20260620 11:46:42.086377 28919 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 11:46:42.086604 28919 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 11:46:42.091427 28925 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 11:46:42.091513 28919 server_base.cc:1034] running on GCE node
W20260620 11:46:42.091414 28924 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 11:46:42.091646 28927 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 11:46:42.091881 28919 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 11:46:42.092167 28919 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 11:46:42.093335 28919 hybrid_clock.cc:648] HybridClock initialized: now 1781956002093314 us; error 46 us; skew 500 ppm
Jun 20 11:46:42 dist-test-slave-cjc6 krb5kdc[28906](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781956002, etypes {rep=17 tkt=17 ses=17}, kudu/127.6.234.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 11:46:42.099411 28919 init.cc:377] Logged in from keytab as kudu/127.6.234.254@KRBTEST.COM (short username kudu)
I20260620 11:46:42.099902 28919 webserver.cc:460] Webserver started at http://127.6.234.254:41193/ using document root <none> and password file <none>
I20260620 11:46:42.100144 28919 fs_manager.cc:359] Metadata directory not provided
I20260620 11:46:42.100212 28919 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 11:46:42.100365 28919 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 11:46:42.101432 28919 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task1JRg7N/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1781955746769051-7083-0/minicluster-data/master-0/data/instance:
uuid: "90ca905cd64b458fbe16002b8c370072"
format_stamp: "Formatted at 2026-06-20 11:46:42 on dist-test-slave-cjc6"
server_key: "feaf426e58d89f225d59998663a741ef"
server_key_iv: "85ff2cdb2a21d17671d32b6fc9ada307"
server_key_version: "encryptionkey@0"
I20260620 11:46:42.101854 28919 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task1JRg7N/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1781955746769051-7083-0/minicluster-data/master-0/wal/instance:
uuid: "90ca905cd64b458fbe16002b8c370072"
format_stamp: "Formatted at 2026-06-20 11:46:42 on dist-test-slave-cjc6"
server_key: "feaf426e58d89f225d59998663a741ef"
server_key_iv: "85ff2cdb2a21d17671d32b6fc9ada307"
server_key_version: "encryptionkey@0"
I20260620 11:46:42.103180 28919 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.001s sys 0.000s
I20260620 11:46:42.103914 28934 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 11:46:42.104089 28919 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260620 11:46:42.104175 28919 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task1JRg7N/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1781955746769051-7083-0/minicluster-data/master-0/data,/tmp/dist-test-task1JRg7N/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1781955746769051-7083-0/minicluster-data/master-0/wal
uuid: "90ca905cd64b458fbe16002b8c370072"
format_stamp: "Formatted at 2026-06-20 11:46:42 on dist-test-slave-cjc6"
server_key: "feaf426e58d89f225d59998663a741ef"
server_key_iv: "85ff2cdb2a21d17671d32b6fc9ada307"
server_key_version: "encryptionkey@0"
I20260620 11:46:42.104261 28919 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task1JRg7N/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1781955746769051-7083-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task1JRg7N/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1781955746769051-7083-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task1JRg7N/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1781955746769051-7083-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 11:46:42.124569 28919 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 11:46:42.126633 28919 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 11:46:42.126794 28919 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 11:46:42.130728 28919 rpc_server.cc:307] RPC server started. Bound to: 127.6.234.254:37845
I20260620 11:46:42.130787 28986 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.6.234.254:37845 every 8 connection(s)
I20260620 11:46:42.131117 28919 server_base.cc:1166] Dumped server information to /tmp/dist-test-task1JRg7N/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1781955746769051-7083-0/minicluster-data/master-0/data/info.pb
I20260620 11:46:42.132077 28987 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 11:46:42.134490 28987 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 90ca905cd64b458fbe16002b8c370072: Bootstrap starting.
I20260620 11:46:42.135067 28987 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 90ca905cd64b458fbe16002b8c370072: Neither blocks nor log segments found. Creating new log.
I20260620 11:46:42.135352 28987 log.cc:824] T 00000000000000000000000000000000 P 90ca905cd64b458fbe16002b8c370072: Log is configured to *not* fsync() on all Append() calls
I20260620 11:46:42.135614 7083 external_mini_cluster.cc:1388] Started /tmp/dist-test-task1JRg7N/build/release/bin/kudu as pid 28919
I20260620 11:46:42.135738 7083 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task1JRg7N/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1781955746769051-7083-0/minicluster-data/master-0/wal/instance
I20260620 11:46:42.135928 7083 external_mini_cluster.cc:1428] Setting key d485684472f2b5087773b3ac498d6bc5
I20260620 11:46:42.136116 28987 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 90ca905cd64b458fbe16002b8c370072: No bootstrap required, opened a new log
I20260620 11:46:42.138334 28987 raft_consensus.cc:348] T 00000000000000000000000000000000 P 90ca905cd64b458fbe16002b8c370072 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "90ca905cd64b458fbe16002b8c370072" member_type: VOTER last_known_addr { host: "127.6.234.254" port: 37845 } }
I20260620 11:46:42.138511 28987 raft_consensus.cc:374] T 00000000000000000000000000000000 P 90ca905cd64b458fbe16002b8c370072 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 11:46:42.138556 28987 raft_consensus.cc:729] T 00000000000000000000000000000000 P 90ca905cd64b458fbe16002b8c370072 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 90ca905cd64b458fbe16002b8c370072, State: Initialized, Role: FOLLOWER
I20260620 11:46:42.138705 28987 consensus_queue.cc:260] T 00000000000000000000000000000000 P 90ca905cd64b458fbe16002b8c370072 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "90ca905cd64b458fbe16002b8c370072" member_type: VOTER last_known_addr { host: "127.6.234.254" port: 37845 } }
I20260620 11:46:42.138783 28987 raft_consensus.cc:388] T 00000000000000000000000000000000 P 90ca905cd64b458fbe16002b8c370072 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260620 11:46:42.138813 28987 raft_consensus.cc:482] T 00000000000000000000000000000000 P 90ca905cd64b458fbe16002b8c370072 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260620 11:46:42.138856 28987 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 90ca905cd64b458fbe16002b8c370072 [term 0 FOLLOWER]: Advancing to term 1
I20260620 11:46:42.139511 28987 raft_consensus.cc:504] T 00000000000000000000000000000000 P 90ca905cd64b458fbe16002b8c370072 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "90ca905cd64b458fbe16002b8c370072" member_type: VOTER last_known_addr { host: "127.6.234.254" port: 37845 } }
I20260620 11:46:42.139616 28987 leader_election.cc:302] T 00000000000000000000000000000000 P 90ca905cd64b458fbe16002b8c370072 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 90ca905cd64b458fbe16002b8c370072; no voters:
I20260620 11:46:42.139757 28987 leader_election.cc:288] T 00000000000000000000000000000000 P 90ca905cd64b458fbe16002b8c370072 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260620 11:46:42.139823 28992 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 90ca905cd64b458fbe16002b8c370072 [term 1 FOLLOWER]: Leader election won for term 1
I20260620 11:46:42.139976 28992 raft_consensus.cc:686] T 00000000000000000000000000000000 P 90ca905cd64b458fbe16002b8c370072 [term 1 LEADER]: Becoming Leader. State: Replica: 90ca905cd64b458fbe16002b8c370072, State: Running, Role: LEADER
I20260620 11:46:42.140144 28992 consensus_queue.cc:237] T 00000000000000000000000000000000 P 90ca905cd64b458fbe16002b8c370072 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "90ca905cd64b458fbe16002b8c370072" member_type: VOTER last_known_addr { host: "127.6.234.254" port: 37845 } }
I20260620 11:46:42.139978 28987 sys_catalog.cc:564] T 00000000000000000000000000000000 P 90ca905cd64b458fbe16002b8c370072 [sys.catalog]: configured and running, proceeding with master startup.
I20260620 11:46:42.140496 28994 sys_catalog.cc:455] T 00000000000000000000000000000000 P 90ca905cd64b458fbe16002b8c370072 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "90ca905cd64b458fbe16002b8c370072" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "90ca905cd64b458fbe16002b8c370072" member_type: VOTER last_known_addr { host: "127.6.234.254" port: 37845 } } }
I20260620 11:46:42.140502 28993 sys_catalog.cc:455] T 00000000000000000000000000000000 P 90ca905cd64b458fbe16002b8c370072 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 90ca905cd64b458fbe16002b8c370072. Latest consensus state: current_term: 1 leader_uuid: "90ca905cd64b458fbe16002b8c370072" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "90ca905cd64b458fbe16002b8c370072" member_type: VOTER last_known_addr { host: "127.6.234.254" port: 37845 } } }
I20260620 11:46:42.140650 28994 sys_catalog.cc:458] T 00000000000000000000000000000000 P 90ca905cd64b458fbe16002b8c370072 [sys.catalog]: This master's current role is: LEADER
I20260620 11:46:42.140658 28993 sys_catalog.cc:458] T 00000000000000000000000000000000 P 90ca905cd64b458fbe16002b8c370072 [sys.catalog]: This master's current role is: LEADER
I20260620 11:46:42.141439 28996 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260620 11:46:42.141991 28996 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260620 11:46:42.143491 28996 catalog_manager.cc:1347] Generated new cluster ID: 9a6f931939124f15bd10d1d929f95a23
I20260620 11:46:42.143546 28996 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
Jun 20 11:46:42 dist-test-slave-cjc6 krb5kdc[28906](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781956001, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.6.234.254@KRBTEST.COM
I20260620 11:46:42.148397 28991 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 11:46:42.136447 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52971 (local address 127.6.234.254:37845)
0620 11:46:42.136661 (+ 214us) server_negotiation.cc:206] Beginning negotiation
0620 11:46:42.136670 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0620 11:46:42.136705 (+ 35us) server_negotiation.cc:407] Connection header received
0620 11:46:42.137483 (+ 778us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 11:46:42.137493 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 11:46:42.137555 (+ 62us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 11:46:42.137754 (+ 199us) server_negotiation.cc:226] Negotiated authn=SASL
0620 11:46:42.138324 (+ 570us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 11:46:42.139994 (+ 1670us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 11:46:42.141128 (+ 1134us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 11:46:42.141762 (+ 634us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 11:46:42.141868 (+ 106us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 11:46:42.144631 (+ 2763us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 11:46:42.144635 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 11:46:42.144635 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 11:46:42.144647 (+ 12us) server_negotiation.cc:922] calling sasl_server_start()
0620 11:46:42.146667 (+ 2020us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 11:46:42.147054 (+ 387us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 11:46:42.147057 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 11:46:42.147058 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 11:46:42.147166 (+ 108us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 11:46:42.147365 (+ 199us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 11:46:42.147366 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 11:46:42.147366 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 11:46:42.147678 (+ 312us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 11:46:42.147857 (+ 179us) server_negotiation.cc:1035] Waiting for connection context
0620 11:46:42.147925 (+ 68us) server_negotiation.cc:299] Negotiation successful
0620 11:46:42.148057 (+ 132us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":145,"thread_start_us":97,"threads_started":1}
I20260620 11:46:42.149438 7083 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260620 11:46:42.158217 28991 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 11:46:42.151008 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37513 (local address 127.6.234.254:37845)
0620 11:46:42.151051 (+ 43us) server_negotiation.cc:206] Beginning negotiation
0620 11:46:42.151053 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 11:46:42.151122 (+ 69us) server_negotiation.cc:407] Connection header received
0620 11:46:42.151215 (+ 93us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 11:46:42.151215 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 11:46:42.151225 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 11:46:42.151253 (+ 28us) server_negotiation.cc:226] Negotiated authn=SASL
0620 11:46:42.151947 (+ 694us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 11:46:42.152695 (+ 748us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 11:46:42.153726 (+ 1031us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 11:46:42.154062 (+ 336us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 11:46:42.154103 (+ 41us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 11:46:42.155503 (+ 1400us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 11:46:42.155504 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 11:46:42.155504 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 11:46:42.155511 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0620 11:46:42.157016 (+ 1505us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 11:46:42.157393 (+ 377us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 11:46:42.157394 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 11:46:42.157395 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 11:46:42.157445 (+ 50us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 11:46:42.157632 (+ 187us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 11:46:42.157633 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 11:46:42.157633 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 11:46:42.157770 (+ 137us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 11:46:42.157848 (+ 78us) server_negotiation.cc:1035] Waiting for connection context
0620 11:46:42.158052 (+ 204us) server_negotiation.cc:299] Negotiation successful
0620 11:46:42.158125 (+ 73us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":24}
I20260620 11:46:42.161491 28996 catalog_manager.cc:1370] Generated new certificate authority record
I20260620 11:46:42.162065 28996 catalog_manager.cc:1504] Loading token signing keys...
I20260620 11:46:42.166483 28996 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 90ca905cd64b458fbe16002b8c370072: Generated new TSK 0
I20260620 11:46:42.166687 28996 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260620 11:46:42.168989 7083 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task1JRg7N/build/release/bin/kudu with pid 28919
2026-06-20T11:46:42Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/15 (1323 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/16
Loading random data
Initializing database '/tmp/dist-test-task1JRg7N/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1781955746769051-7083-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 20 11:46:42 dist-test-slave-cjc6 krb5kdc[29018](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 20 11:46:42 dist-test-slave-cjc6 krb5kdc[29018](info): set up 2 sockets
Jun 20 11:46:42 dist-test-slave-cjc6 krb5kdc[29018](info): commencing operation
krb5kdc: starting...
W20260620 11:46:43.197927 7083 mini_kdc.cc:121] Time spent starting KDC: real 1.011s user 0.000s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task1JRg7N/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1781955746769051-7083-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task1JRg7N/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1781955746769051-7083-0/krb5kdc/test-user.keytab.
Jun 20 11:46:43 dist-test-slave-cjc6 krb5kdc[29018](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781956003, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-20T11:46:43Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-20T11:46:43Z Disabled control of system clock
WARNING: no policy specified for kudu/127.6.234.254@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.6.234.254@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.6.234.254 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task1JRg7N/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1781955746769051-7083-0/krb5kdc/kudu_127.6.234.254.keytab.
Entry for principal kudu/127.6.234.254 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task1JRg7N/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1781955746769051-7083-0/krb5kdc/kudu_127.6.234.254.keytab.
I20260620 11:46:43.313939 7083 external_mini_cluster.cc:1326] Running /tmp/dist-test-task1JRg7N/build/release/bin/kudu
/tmp/dist-test-task1JRg7N/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task1JRg7N/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1781955746769051-7083-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task1JRg7N/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1781955746769051-7083-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task1JRg7N/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1781955746769051-7083-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task1JRg7N/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1781955746769051-7083-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.6.234.254:38319
--webserver_interface=127.6.234.254
--webserver_port=0
--keytab_file=/tmp/dist-test-task1JRg7N/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1781955746769051-7083-0/krb5kdc/kudu_127.6.234.254.keytab
--principal=kudu/127.6.234.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.6.234.212:33283
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.6.234.254:38319
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=required
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-task1JRg7N/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1781955746769051-7083-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task1JRg7N/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1781955746769051-7083-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task1JRg7N/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1781955746769051-7083-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 11:46:43.395277 29031 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 11:46:43.395493 29031 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 11:46:43.395517 29031 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 11:46:43.396747 29031 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 11:46:43.396793 29031 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 11:46:43.396812 29031 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 11:46:43.396826 29031 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 11:46:43.396890 29031 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 11:46:43.398501 29031 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.6.234.212:33283
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task1JRg7N/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1781955746769051-7083-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task1JRg7N/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1781955746769051-7083-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.6.234.254:38319
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task1JRg7N/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1781955746769051-7083-0/krb5kdc/kudu_127.6.234.254.keytab
--principal=kudu/127.6.234.254
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.6.234.254:38319
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task1JRg7N/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1781955746769051-7083-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.6.234.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.29031
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task1JRg7N/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1781955746769051-7083-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 20 Jun 2026 10:32:37 UTC on ccycloud.quasar-bgyofv.root.comops.site
build id 4457173
I20260620 11:46:43.398766 29031 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 11:46:43.398999 29031 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 11:46:43.403858 29039 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 11:46:43.403868 29037 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 11:46:43.403864 29036 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 11:46:43.404037 29031 server_base.cc:1034] running on GCE node
I20260620 11:46:43.404279 29031 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 11:46:43.404561 29031 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 11:46:43.405767 29031 hybrid_clock.cc:648] HybridClock initialized: now 1781956003405751 us; error 27 us; skew 500 ppm
Jun 20 11:46:43 dist-test-slave-cjc6 krb5kdc[29018](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781956003, etypes {rep=17 tkt=17 ses=17}, kudu/127.6.234.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 11:46:43.408254 29031 init.cc:377] Logged in from keytab as kudu/127.6.234.254@KRBTEST.COM (short username kudu)
I20260620 11:46:43.408670 29031 webserver.cc:460] Webserver started at http://127.6.234.254:40721/ using document root <none> and password file <none>
I20260620 11:46:43.408929 29031 fs_manager.cc:359] Metadata directory not provided
I20260620 11:46:43.408998 29031 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 11:46:43.409147 29031 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 11:46:43.410285 29031 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task1JRg7N/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1781955746769051-7083-0/minicluster-data/master-0/data/instance:
uuid: "710972ce9d5340289a66c0a05ae9a6f7"
format_stamp: "Formatted at 2026-06-20 11:46:43 on dist-test-slave-cjc6"
server_key: "de9c88b8b514cbbc32bb84c430844747"
server_key_iv: "88625d3660824d7b027436430fc6aa55"
server_key_version: "encryptionkey@0"
I20260620 11:46:43.410641 29031 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task1JRg7N/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1781955746769051-7083-0/minicluster-data/master-0/wal/instance:
uuid: "710972ce9d5340289a66c0a05ae9a6f7"
format_stamp: "Formatted at 2026-06-20 11:46:43 on dist-test-slave-cjc6"
server_key: "de9c88b8b514cbbc32bb84c430844747"
server_key_iv: "88625d3660824d7b027436430fc6aa55"
server_key_version: "encryptionkey@0"
I20260620 11:46:43.412031 29031 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260620 11:46:43.412760 29046 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 11:46:43.412950 29031 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260620 11:46:43.413058 29031 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task1JRg7N/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1781955746769051-7083-0/minicluster-data/master-0/data,/tmp/dist-test-task1JRg7N/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1781955746769051-7083-0/minicluster-data/master-0/wal
uuid: "710972ce9d5340289a66c0a05ae9a6f7"
format_stamp: "Formatted at 2026-06-20 11:46:43 on dist-test-slave-cjc6"
server_key: "de9c88b8b514cbbc32bb84c430844747"
server_key_iv: "88625d3660824d7b027436430fc6aa55"
server_key_version: "encryptionkey@0"
I20260620 11:46:43.413130 29031 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task1JRg7N/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1781955746769051-7083-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task1JRg7N/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1781955746769051-7083-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task1JRg7N/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1781955746769051-7083-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 11:46:43.424036 29031 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 11:46:43.426288 29031 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 11:46:43.426467 29031 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 11:46:43.430444 29031 rpc_server.cc:307] RPC server started. Bound to: 127.6.234.254:38319
I20260620 11:46:43.430478 29098 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.6.234.254:38319 every 8 connection(s)
I20260620 11:46:43.430835 29031 server_base.cc:1166] Dumped server information to /tmp/dist-test-task1JRg7N/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1781955746769051-7083-0/minicluster-data/master-0/data/info.pb
I20260620 11:46:43.431664 29099 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 11:46:43.433938 29099 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 710972ce9d5340289a66c0a05ae9a6f7: Bootstrap starting.
I20260620 11:46:43.434511 29099 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 710972ce9d5340289a66c0a05ae9a6f7: Neither blocks nor log segments found. Creating new log.
I20260620 11:46:43.434796 29099 log.cc:824] T 00000000000000000000000000000000 P 710972ce9d5340289a66c0a05ae9a6f7: Log is configured to *not* fsync() on all Append() calls
I20260620 11:46:43.435461 29099 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 710972ce9d5340289a66c0a05ae9a6f7: No bootstrap required, opened a new log
I20260620 11:46:43.437008 29099 raft_consensus.cc:348] T 00000000000000000000000000000000 P 710972ce9d5340289a66c0a05ae9a6f7 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "710972ce9d5340289a66c0a05ae9a6f7" member_type: VOTER last_known_addr { host: "127.6.234.254" port: 38319 } }
I20260620 11:46:43.437143 29099 raft_consensus.cc:374] T 00000000000000000000000000000000 P 710972ce9d5340289a66c0a05ae9a6f7 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 11:46:43.437166 29099 raft_consensus.cc:729] T 00000000000000000000000000000000 P 710972ce9d5340289a66c0a05ae9a6f7 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 710972ce9d5340289a66c0a05ae9a6f7, State: Initialized, Role: FOLLOWER
I20260620 11:46:43.437362 29099 consensus_queue.cc:260] T 00000000000000000000000000000000 P 710972ce9d5340289a66c0a05ae9a6f7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "710972ce9d5340289a66c0a05ae9a6f7" member_type: VOTER last_known_addr { host: "127.6.234.254" port: 38319 } }
I20260620 11:46:43.437444 29099 raft_consensus.cc:388] T 00000000000000000000000000000000 P 710972ce9d5340289a66c0a05ae9a6f7 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260620 11:46:43.437486 29099 raft_consensus.cc:482] T 00000000000000000000000000000000 P 710972ce9d5340289a66c0a05ae9a6f7 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260620 11:46:43.437533 29099 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 710972ce9d5340289a66c0a05ae9a6f7 [term 0 FOLLOWER]: Advancing to term 1
I20260620 11:46:43.438426 29099 raft_consensus.cc:504] T 00000000000000000000000000000000 P 710972ce9d5340289a66c0a05ae9a6f7 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "710972ce9d5340289a66c0a05ae9a6f7" member_type: VOTER last_known_addr { host: "127.6.234.254" port: 38319 } }
I20260620 11:46:43.438535 29099 leader_election.cc:302] T 00000000000000000000000000000000 P 710972ce9d5340289a66c0a05ae9a6f7 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 710972ce9d5340289a66c0a05ae9a6f7; no voters:
I20260620 11:46:43.438707 29099 leader_election.cc:288] T 00000000000000000000000000000000 P 710972ce9d5340289a66c0a05ae9a6f7 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260620 11:46:43.438731 29102 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 710972ce9d5340289a66c0a05ae9a6f7 [term 1 FOLLOWER]: Leader election won for term 1
I20260620 11:46:43.438736 7083 external_mini_cluster.cc:1388] Started /tmp/dist-test-task1JRg7N/build/release/bin/kudu as pid 29031
I20260620 11:46:43.438859 29102 raft_consensus.cc:686] T 00000000000000000000000000000000 P 710972ce9d5340289a66c0a05ae9a6f7 [term 1 LEADER]: Becoming Leader. State: Replica: 710972ce9d5340289a66c0a05ae9a6f7, State: Running, Role: LEADER
I20260620 11:46:43.438879 7083 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task1JRg7N/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1781955746769051-7083-0/minicluster-data/master-0/wal/instance
I20260620 11:46:43.438969 29099 sys_catalog.cc:564] T 00000000000000000000000000000000 P 710972ce9d5340289a66c0a05ae9a6f7 [sys.catalog]: configured and running, proceeding with master startup.
I20260620 11:46:43.439023 29102 consensus_queue.cc:237] T 00000000000000000000000000000000 P 710972ce9d5340289a66c0a05ae9a6f7 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "710972ce9d5340289a66c0a05ae9a6f7" member_type: VOTER last_known_addr { host: "127.6.234.254" port: 38319 } }
I20260620 11:46:43.439059 7083 external_mini_cluster.cc:1428] Setting key f4b6a2929f3ee1961891aeee1aae6d6d
I20260620 11:46:43.439437 29103 sys_catalog.cc:455] T 00000000000000000000000000000000 P 710972ce9d5340289a66c0a05ae9a6f7 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 710972ce9d5340289a66c0a05ae9a6f7. Latest consensus state: current_term: 1 leader_uuid: "710972ce9d5340289a66c0a05ae9a6f7" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "710972ce9d5340289a66c0a05ae9a6f7" member_type: VOTER last_known_addr { host: "127.6.234.254" port: 38319 } } }
I20260620 11:46:43.439527 29103 sys_catalog.cc:458] T 00000000000000000000000000000000 P 710972ce9d5340289a66c0a05ae9a6f7 [sys.catalog]: This master's current role is: LEADER
I20260620 11:46:43.439774 29104 sys_catalog.cc:455] T 00000000000000000000000000000000 P 710972ce9d5340289a66c0a05ae9a6f7 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "710972ce9d5340289a66c0a05ae9a6f7" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "710972ce9d5340289a66c0a05ae9a6f7" member_type: VOTER last_known_addr { host: "127.6.234.254" port: 38319 } } }
I20260620 11:46:43.440078 29104 sys_catalog.cc:458] T 00000000000000000000000000000000 P 710972ce9d5340289a66c0a05ae9a6f7 [sys.catalog]: This master's current role is: LEADER
W20260620 11:46:43.440764 29120 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 710972ce9d5340289a66c0a05ae9a6f7: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260620 11:46:43.440832 29120 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260620 11:46:43.440930 29111 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260620 11:46:43.441099 29111 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260620 11:46:43.442589 29111 catalog_manager.cc:1347] Generated new cluster ID: a1d49062e1fa4009ac90d9a4ddf9649b
I20260620 11:46:43.442651 29111 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260620 11:46:43.444991 29113 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 11:46:43.439668 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55881 (local address 127.6.234.254:38319)
0620 11:46:43.439913 (+ 245us) server_negotiation.cc:206] Beginning negotiation
0620 11:46:43.439923 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0620 11:46:43.439957 (+ 34us) server_negotiation.cc:407] Connection header received
0620 11:46:43.440103 (+ 146us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 11:46:43.440110 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 11:46:43.440167 (+ 57us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 11:46:43.440316 (+ 149us) server_negotiation.cc:226] Negotiated authn=SASL
0620 11:46:43.440652 (+ 336us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 11:46:43.442084 (+ 1432us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 11:46:43.443098 (+ 1014us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 11:46:43.443511 (+ 413us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 11:46:43.443630 (+ 119us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 11:46:43.444347 (+ 717us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 11:46:43.444351 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 11:46:43.444351 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 11:46:43.444525 (+ 174us) server_negotiation.cc:922] calling sasl_server_start()
0620 11:46:43.444555 (+ 30us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 11:46:43.444566 (+ 11us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 11:46:43.444637 (+ 71us) server_negotiation.cc:1035] Waiting for connection context
0620 11:46:43.444711 (+ 74us) server_negotiation.cc:299] Negotiation successful
0620 11:46:43.444781 (+ 70us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":194,"thread_start_us":76,"threads_started":1}
I20260620 11:46:43.445968 7083 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260620 11:46:43.449990 29113 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 11:46:43.447418 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33723 (local address 127.6.234.254:38319)
0620 11:46:43.447453 (+ 35us) server_negotiation.cc:206] Beginning negotiation
0620 11:46:43.447455 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 11:46:43.447462 (+ 7us) server_negotiation.cc:407] Connection header received
0620 11:46:43.447493 (+ 31us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 11:46:43.447494 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 11:46:43.447502 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 11:46:43.447547 (+ 45us) server_negotiation.cc:226] Negotiated authn=SASL
0620 11:46:43.447658 (+ 111us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 11:46:43.448115 (+ 457us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 11:46:43.448798 (+ 683us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 11:46:43.449003 (+ 205us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 11:46:43.449086 (+ 83us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 11:46:43.449750 (+ 664us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 11:46:43.449752 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 11:46:43.449758 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 11:46:43.449763 (+ 5us) server_negotiation.cc:922] calling sasl_server_start()
0620 11:46:43.449776 (+ 13us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 11:46:43.449781 (+ 5us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 11:46:43.449834 (+ 53us) server_negotiation.cc:1035] Waiting for connection context
0620 11:46:43.449893 (+ 59us) server_negotiation.cc:299] Negotiation successful
0620 11:46:43.449915 (+ 22us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":19}
I20260620 11:46:43.457199 29111 catalog_manager.cc:1370] Generated new certificate authority record
I20260620 11:46:43.457957 29111 catalog_manager.cc:1504] Loading token signing keys...
I20260620 11:46:43.463697 29111 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 710972ce9d5340289a66c0a05ae9a6f7: Generated new TSK 0
I20260620 11:46:43.463907 29111 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260620 11:46:43.471521 7083 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task1JRg7N/build/release/bin/kudu with pid 29031
2026-06-20T11:46:43Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/16 (1302 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/17
Loading random data
Initializing database '/tmp/dist-test-task1JRg7N/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1781955746769051-7083-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 20 11:46:43 dist-test-slave-cjc6 krb5kdc[29130](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 20 11:46:43 dist-test-slave-cjc6 krb5kdc[29130](info): set up 2 sockets
Jun 20 11:46:43 dist-test-slave-cjc6 krb5kdc[29130](info): commencing operation
krb5kdc: starting...
W20260620 11:46:44.493948 7083 mini_kdc.cc:121] Time spent starting KDC: real 1.004s user 0.000s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task1JRg7N/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1781955746769051-7083-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task1JRg7N/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1781955746769051-7083-0/krb5kdc/test-user.keytab.
Jun 20 11:46:44 dist-test-slave-cjc6 krb5kdc[29130](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781956004, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-20T11:46:44Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-20T11:46:44Z Disabled control of system clock
WARNING: no policy specified for kudu/127.6.234.254@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.6.234.254@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.6.234.254 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task1JRg7N/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1781955746769051-7083-0/krb5kdc/kudu_127.6.234.254.keytab.
Entry for principal kudu/127.6.234.254 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task1JRg7N/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1781955746769051-7083-0/krb5kdc/kudu_127.6.234.254.keytab.
I20260620 11:46:44.612160 7083 external_mini_cluster.cc:1326] Running /tmp/dist-test-task1JRg7N/build/release/bin/kudu
/tmp/dist-test-task1JRg7N/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task1JRg7N/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1781955746769051-7083-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task1JRg7N/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1781955746769051-7083-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task1JRg7N/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1781955746769051-7083-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task1JRg7N/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1781955746769051-7083-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.6.234.254:45785
--webserver_interface=127.6.234.254
--webserver_port=0
--keytab_file=/tmp/dist-test-task1JRg7N/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1781955746769051-7083-0/krb5kdc/kudu_127.6.234.254.keytab
--principal=kudu/127.6.234.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.6.234.212:33617
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.6.234.254:45785
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=disabled
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-task1JRg7N/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1781955746769051-7083-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task1JRg7N/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1781955746769051-7083-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task1JRg7N/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1781955746769051-7083-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 11:46:44.694216 29143 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 11:46:44.694438 29143 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 11:46:44.694463 29143 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 11:46:44.695644 29143 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 11:46:44.695690 29143 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 11:46:44.695705 29143 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 11:46:44.695717 29143 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 11:46:44.695761 29143 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 11:46:44.697327 29143 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.6.234.212:33617
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task1JRg7N/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1781955746769051-7083-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task1JRg7N/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1781955746769051-7083-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.6.234.254:45785
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task1JRg7N/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1781955746769051-7083-0/krb5kdc/kudu_127.6.234.254.keytab
--principal=kudu/127.6.234.254
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.6.234.254:45785
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task1JRg7N/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1781955746769051-7083-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.6.234.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.29143
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task1JRg7N/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1781955746769051-7083-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 20 Jun 2026 10:32:37 UTC on ccycloud.quasar-bgyofv.root.comops.site
build id 4457173
I20260620 11:46:44.697584 29143 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 11:46:44.697840 29143 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 11:46:44.702850 29149 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 11:46:44.702850 29148 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 11:46:44.703106 29143 server_base.cc:1034] running on GCE node
W20260620 11:46:44.703171 29151 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 11:46:44.703413 29143 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 11:46:44.703708 29143 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 11:46:44.704873 29143 hybrid_clock.cc:648] HybridClock initialized: now 1781956004704861 us; error 52 us; skew 500 ppm
Jun 20 11:46:44 dist-test-slave-cjc6 krb5kdc[29130](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781956004, etypes {rep=17 tkt=17 ses=17}, kudu/127.6.234.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 11:46:44.707435 29143 init.cc:377] Logged in from keytab as kudu/127.6.234.254@KRBTEST.COM (short username kudu)
I20260620 11:46:44.707881 29143 webserver.cc:460] Webserver started at http://127.6.234.254:46549/ using document root <none> and password file <none>
I20260620 11:46:44.708112 29143 fs_manager.cc:359] Metadata directory not provided
I20260620 11:46:44.708181 29143 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 11:46:44.708341 29143 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 11:46:44.709439 29143 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task1JRg7N/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1781955746769051-7083-0/minicluster-data/master-0/data/instance:
uuid: "b24ef0f227584f029b9be2ec37fa41de"
format_stamp: "Formatted at 2026-06-20 11:46:44 on dist-test-slave-cjc6"
server_key: "048914cf3cd93bdf8348bbdd366e051c"
server_key_iv: "096c52563cd826c1c4f077b21a9134bb"
server_key_version: "encryptionkey@0"
I20260620 11:46:44.709841 29143 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task1JRg7N/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1781955746769051-7083-0/minicluster-data/master-0/wal/instance:
uuid: "b24ef0f227584f029b9be2ec37fa41de"
format_stamp: "Formatted at 2026-06-20 11:46:44 on dist-test-slave-cjc6"
server_key: "048914cf3cd93bdf8348bbdd366e051c"
server_key_iv: "096c52563cd826c1c4f077b21a9134bb"
server_key_version: "encryptionkey@0"
I20260620 11:46:44.711210 29143 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.001s
I20260620 11:46:44.711941 29158 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 11:46:44.712108 29143 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.001s
I20260620 11:46:44.712224 29143 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task1JRg7N/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1781955746769051-7083-0/minicluster-data/master-0/data,/tmp/dist-test-task1JRg7N/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1781955746769051-7083-0/minicluster-data/master-0/wal
uuid: "b24ef0f227584f029b9be2ec37fa41de"
format_stamp: "Formatted at 2026-06-20 11:46:44 on dist-test-slave-cjc6"
server_key: "048914cf3cd93bdf8348bbdd366e051c"
server_key_iv: "096c52563cd826c1c4f077b21a9134bb"
server_key_version: "encryptionkey@0"
I20260620 11:46:44.712306 29143 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task1JRg7N/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1781955746769051-7083-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task1JRg7N/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1781955746769051-7083-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task1JRg7N/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1781955746769051-7083-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 11:46:44.715677 29143 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 11:46:44.718070 29143 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 11:46:44.718235 29143 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 11:46:44.722126 29143 rpc_server.cc:307] RPC server started. Bound to: 127.6.234.254:45785
I20260620 11:46:44.722157 29210 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.6.234.254:45785 every 8 connection(s)
I20260620 11:46:44.722496 29143 server_base.cc:1166] Dumped server information to /tmp/dist-test-task1JRg7N/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1781955746769051-7083-0/minicluster-data/master-0/data/info.pb
I20260620 11:46:44.723439 29211 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 11:46:44.725759 29211 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P b24ef0f227584f029b9be2ec37fa41de: Bootstrap starting.
I20260620 11:46:44.726356 29211 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P b24ef0f227584f029b9be2ec37fa41de: Neither blocks nor log segments found. Creating new log.
I20260620 11:46:44.726643 29211 log.cc:824] T 00000000000000000000000000000000 P b24ef0f227584f029b9be2ec37fa41de: Log is configured to *not* fsync() on all Append() calls
I20260620 11:46:44.726890 7083 external_mini_cluster.cc:1388] Started /tmp/dist-test-task1JRg7N/build/release/bin/kudu as pid 29143
I20260620 11:46:44.727030 7083 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task1JRg7N/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1781955746769051-7083-0/minicluster-data/master-0/wal/instance
I20260620 11:46:44.727237 7083 external_mini_cluster.cc:1428] Setting key 2ea33ee516f311f5a96291f71c442f36
I20260620 11:46:44.727488 29211 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P b24ef0f227584f029b9be2ec37fa41de: No bootstrap required, opened a new log
I20260620 11:46:44.729235 29211 raft_consensus.cc:348] T 00000000000000000000000000000000 P b24ef0f227584f029b9be2ec37fa41de [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b24ef0f227584f029b9be2ec37fa41de" member_type: VOTER last_known_addr { host: "127.6.234.254" port: 45785 } }
I20260620 11:46:44.729398 29211 raft_consensus.cc:374] T 00000000000000000000000000000000 P b24ef0f227584f029b9be2ec37fa41de [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 11:46:44.729460 29211 raft_consensus.cc:729] T 00000000000000000000000000000000 P b24ef0f227584f029b9be2ec37fa41de [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b24ef0f227584f029b9be2ec37fa41de, State: Initialized, Role: FOLLOWER
I20260620 11:46:44.729596 29211 consensus_queue.cc:260] T 00000000000000000000000000000000 P b24ef0f227584f029b9be2ec37fa41de [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b24ef0f227584f029b9be2ec37fa41de" member_type: VOTER last_known_addr { host: "127.6.234.254" port: 45785 } }
I20260620 11:46:44.729728 29211 raft_consensus.cc:388] T 00000000000000000000000000000000 P b24ef0f227584f029b9be2ec37fa41de [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260620 11:46:44.729764 29211 raft_consensus.cc:482] T 00000000000000000000000000000000 P b24ef0f227584f029b9be2ec37fa41de [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260620 11:46:44.729815 29211 raft_consensus.cc:3037] T 00000000000000000000000000000000 P b24ef0f227584f029b9be2ec37fa41de [term 0 FOLLOWER]: Advancing to term 1
I20260620 11:46:44.730584 29215 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 11:46:44.727773 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37913 (local address 127.6.234.254:45785)
0620 11:46:44.727990 (+ 217us) server_negotiation.cc:206] Beginning negotiation
0620 11:46:44.728000 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0620 11:46:44.728042 (+ 42us) server_negotiation.cc:407] Connection header received
0620 11:46:44.729226 (+ 1184us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 11:46:44.729234 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 11:46:44.729266 (+ 32us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 11:46:44.729420 (+ 154us) server_negotiation.cc:226] Negotiated authn=SASL
0620 11:46:44.730141 (+ 721us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 11:46:44.730145 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 11:46:44.730145 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 11:46:44.730150 (+ 5us) server_negotiation.cc:922] calling sasl_server_start()
0620 11:46:44.730202 (+ 52us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 11:46:44.730214 (+ 12us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 11:46:44.730293 (+ 79us) server_negotiation.cc:1035] Waiting for connection context
0620 11:46:44.730360 (+ 67us) server_negotiation.cc:299] Negotiation successful
0620 11:46:44.730404 (+ 44us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":144,"thread_start_us":82,"threads_started":1}
I20260620 11:46:44.730834 29211 raft_consensus.cc:504] T 00000000000000000000000000000000 P b24ef0f227584f029b9be2ec37fa41de [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b24ef0f227584f029b9be2ec37fa41de" member_type: VOTER last_known_addr { host: "127.6.234.254" port: 45785 } }
I20260620 11:46:44.730971 29211 leader_election.cc:302] T 00000000000000000000000000000000 P b24ef0f227584f029b9be2ec37fa41de [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: b24ef0f227584f029b9be2ec37fa41de; no voters:
I20260620 11:46:44.731134 29211 leader_election.cc:288] T 00000000000000000000000000000000 P b24ef0f227584f029b9be2ec37fa41de [CANDIDATE]: Term 1 election: Requested vote from peers
I20260620 11:46:44.731177 29216 raft_consensus.cc:2781] T 00000000000000000000000000000000 P b24ef0f227584f029b9be2ec37fa41de [term 1 FOLLOWER]: Leader election won for term 1
I20260620 11:46:44.731348 29216 raft_consensus.cc:686] T 00000000000000000000000000000000 P b24ef0f227584f029b9be2ec37fa41de [term 1 LEADER]: Becoming Leader. State: Replica: b24ef0f227584f029b9be2ec37fa41de, State: Running, Role: LEADER
I20260620 11:46:44.731463 29211 sys_catalog.cc:564] T 00000000000000000000000000000000 P b24ef0f227584f029b9be2ec37fa41de [sys.catalog]: configured and running, proceeding with master startup.
I20260620 11:46:44.731482 29216 consensus_queue.cc:237] T 00000000000000000000000000000000 P b24ef0f227584f029b9be2ec37fa41de [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b24ef0f227584f029b9be2ec37fa41de" member_type: VOTER last_known_addr { host: "127.6.234.254" port: 45785 } }
I20260620 11:46:44.731678 7083 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260620 11:46:44.731912 29216 sys_catalog.cc:455] T 00000000000000000000000000000000 P b24ef0f227584f029b9be2ec37fa41de [sys.catalog]: SysCatalogTable state changed. Reason: New leader b24ef0f227584f029b9be2ec37fa41de. Latest consensus state: current_term: 1 leader_uuid: "b24ef0f227584f029b9be2ec37fa41de" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b24ef0f227584f029b9be2ec37fa41de" member_type: VOTER last_known_addr { host: "127.6.234.254" port: 45785 } } }
I20260620 11:46:44.731979 29216 sys_catalog.cc:458] T 00000000000000000000000000000000 P b24ef0f227584f029b9be2ec37fa41de [sys.catalog]: This master's current role is: LEADER
I20260620 11:46:44.733013 29218 sys_catalog.cc:455] T 00000000000000000000000000000000 P b24ef0f227584f029b9be2ec37fa41de [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "b24ef0f227584f029b9be2ec37fa41de" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b24ef0f227584f029b9be2ec37fa41de" member_type: VOTER last_known_addr { host: "127.6.234.254" port: 45785 } } }
I20260620 11:46:44.733160 29218 sys_catalog.cc:458] T 00000000000000000000000000000000 P b24ef0f227584f029b9be2ec37fa41de [sys.catalog]: This master's current role is: LEADER
I20260620 11:46:44.733098 29220 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260620 11:46:44.733882 29220 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260620 11:46:44.739017 29220 catalog_manager.cc:1347] Generated new cluster ID: 1629ebaebf0b4db48d20d9b3eb435838
I20260620 11:46:44.739095 29220 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260620 11:46:44.739138 29215 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 11:46:44.733219 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55805 (local address 127.6.234.254:45785)
0620 11:46:44.733256 (+ 37us) server_negotiation.cc:206] Beginning negotiation
0620 11:46:44.733257 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0620 11:46:44.737903 (+ 4646us) server_negotiation.cc:407] Connection header received
0620 11:46:44.737996 (+ 93us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 11:46:44.737997 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 11:46:44.738004 (+ 7us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 11:46:44.738080 (+ 76us) server_negotiation.cc:226] Negotiated authn=SASL
0620 11:46:44.738787 (+ 707us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 11:46:44.738789 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 11:46:44.738789 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 11:46:44.738794 (+ 5us) server_negotiation.cc:922] calling sasl_server_start()
0620 11:46:44.738808 (+ 14us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 11:46:44.738813 (+ 5us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 11:46:44.738872 (+ 59us) server_negotiation.cc:1035] Waiting for connection context
0620 11:46:44.738979 (+ 107us) server_negotiation.cc:299] Negotiation successful
0620 11:46:44.738996 (+ 17us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":18}
I20260620 11:46:44.748421 29220 catalog_manager.cc:1370] Generated new certificate authority record
I20260620 11:46:44.748624 29220 catalog_manager.cc:1504] Loading token signing keys...
I20260620 11:46:44.750705 29220 catalog_manager.cc:5868] T 00000000000000000000000000000000 P b24ef0f227584f029b9be2ec37fa41de: Generated new TSK 0
I20260620 11:46:44.750907 29220 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260620 11:46:44.758244 7083 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task1JRg7N/build/release/bin/kudu with pid 29143
2026-06-20T11:46:44Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/17 (1286 ms)
[----------] 18 tests from AuthTokenIssuingTest (23647 ms total)
[----------] 2 tests from ConnectToFollowerMasterTest
[ RUN ] ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys/0
Loading random data
Initializing database '/tmp/dist-test-task1JRg7N/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781955746769051-7083-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 20 11:46:44 dist-test-slave-cjc6 krb5kdc[29242](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 20 11:46:44 dist-test-slave-cjc6 krb5kdc[29242](info): set up 2 sockets
Jun 20 11:46:44 dist-test-slave-cjc6 krb5kdc[29242](info): commencing operation
krb5kdc: starting...
W20260620 11:46:45.797871 7083 mini_kdc.cc:121] Time spent starting KDC: real 1.022s user 0.003s sys 0.003s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task1JRg7N/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781955746769051-7083-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task1JRg7N/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781955746769051-7083-0/krb5kdc/test-user.keytab.
Jun 20 11:46:45 dist-test-slave-cjc6 krb5kdc[29242](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781956005, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-20T11:46:45Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-20T11:46:45Z Disabled control of system clock
WARNING: no policy specified for kudu/127.6.234.254@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.6.234.254@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.6.234.254 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task1JRg7N/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781955746769051-7083-0/krb5kdc/kudu_127.6.234.254.keytab.
Entry for principal kudu/127.6.234.254 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task1JRg7N/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781955746769051-7083-0/krb5kdc/kudu_127.6.234.254.keytab.
I20260620 11:46:45.914330 7083 external_mini_cluster.cc:1326] Running /tmp/dist-test-task1JRg7N/build/release/bin/kudu
/tmp/dist-test-task1JRg7N/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task1JRg7N/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781955746769051-7083-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task1JRg7N/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781955746769051-7083-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task1JRg7N/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781955746769051-7083-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task1JRg7N/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781955746769051-7083-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.6.234.254:44951
--webserver_interface=127.6.234.254
--webserver_port=0
--keytab_file=/tmp/dist-test-task1JRg7N/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781955746769051-7083-0/krb5kdc/kudu_127.6.234.254.keytab
--principal=kudu/127.6.234.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.6.234.212:36361
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.6.234.254:44951,127.6.234.253:36225,127.6.234.252:37231
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--enable_leader_failure_detection=false
--rpc_authentication=required
--rpc_encryption=optional with env {KRB5CCNAME=/tmp/dist-test-task1JRg7N/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781955746769051-7083-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task1JRg7N/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781955746769051-7083-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task1JRg7N/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781955746769051-7083-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 11:46:45.993752 29255 flags.cc:425] Enabled unsafe flag: --enable_leader_failure_detection=false
W20260620 11:46:45.993973 29255 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 11:46:45.993990 29255 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 11:46:45.994010 29255 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 11:46:45.995204 29255 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 11:46:45.995249 29255 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 11:46:45.995265 29255 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 11:46:45.995276 29255 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 11:46:45.995288 29255 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 11:46:45.996836 29255 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.6.234.212:36361
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--enable_leader_failure_detection=false
--fs_data_dirs=/tmp/dist-test-task1JRg7N/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781955746769051-7083-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task1JRg7N/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781955746769051-7083-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.6.234.254:44951,127.6.234.253:36225,127.6.234.252:37231
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task1JRg7N/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781955746769051-7083-0/krb5kdc/kudu_127.6.234.254.keytab
--principal=kudu/127.6.234.254
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.6.234.254:44951
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task1JRg7N/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781955746769051-7083-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.6.234.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.29255
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task1JRg7N/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781955746769051-7083-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 20 Jun 2026 10:32:37 UTC on ccycloud.quasar-bgyofv.root.comops.site
build id 4457173
I20260620 11:46:45.997112 29255 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 11:46:45.997355 29255 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 11:46:46.002192 29263 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 11:46:46.002200 29261 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 11:46:46.002445 29255 server_base.cc:1034] running on GCE node
W20260620 11:46:46.002192 29260 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 11:46:46.002719 29255 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 11:46:46.003007 29255 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 11:46:46.004190 29255 hybrid_clock.cc:648] HybridClock initialized: now 1781956006004163 us; error 39 us; skew 500 ppm
Jun 20 11:46:46 dist-test-slave-cjc6 krb5kdc[29242](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781956006, etypes {rep=17 tkt=17 ses=17}, kudu/127.6.234.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 11:46:46.006716 29255 init.cc:377] Logged in from keytab as kudu/127.6.234.254@KRBTEST.COM (short username kudu)
I20260620 11:46:46.007145 29255 webserver.cc:460] Webserver started at http://127.6.234.254:38309/ using document root <none> and password file <none>
I20260620 11:46:46.007381 29255 fs_manager.cc:359] Metadata directory not provided
I20260620 11:46:46.007450 29255 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 11:46:46.007602 29255 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 11:46:46.008720 29255 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task1JRg7N/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781955746769051-7083-0/minicluster-data/master-0/data/instance:
uuid: "5f9b7135cde94d50aea81953f2b45385"
format_stamp: "Formatted at 2026-06-20 11:46:46 on dist-test-slave-cjc6"
server_key: "2c7ccf3de2289e35c3adc3e0f66a1b21"
server_key_iv: "37cddab621313ebf147c74003bddf9a7"
server_key_version: "encryptionkey@0"
I20260620 11:46:46.009387 29255 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task1JRg7N/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781955746769051-7083-0/minicluster-data/master-0/wal/instance:
uuid: "5f9b7135cde94d50aea81953f2b45385"
format_stamp: "Formatted at 2026-06-20 11:46:46 on dist-test-slave-cjc6"
server_key: "2c7ccf3de2289e35c3adc3e0f66a1b21"
server_key_iv: "37cddab621313ebf147c74003bddf9a7"
server_key_version: "encryptionkey@0"
I20260620 11:46:46.011059 29255 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.004s sys 0.000s
I20260620 11:46:46.011845 29270 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 11:46:46.012037 29255 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260620 11:46:46.012142 29255 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task1JRg7N/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781955746769051-7083-0/minicluster-data/master-0/data,/tmp/dist-test-task1JRg7N/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781955746769051-7083-0/minicluster-data/master-0/wal
uuid: "5f9b7135cde94d50aea81953f2b45385"
format_stamp: "Formatted at 2026-06-20 11:46:46 on dist-test-slave-cjc6"
server_key: "2c7ccf3de2289e35c3adc3e0f66a1b21"
server_key_iv: "37cddab621313ebf147c74003bddf9a7"
server_key_version: "encryptionkey@0"
I20260620 11:46:46.012233 29255 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task1JRg7N/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781955746769051-7083-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task1JRg7N/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781955746769051-7083-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task1JRg7N/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781955746769051-7083-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 11:46:46.030777 29255 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 11:46:46.033215 29255 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 11:46:46.033403 29255 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 11:46:46.037245 29255 rpc_server.cc:307] RPC server started. Bound to: 127.6.234.254:44951
I20260620 11:46:46.037297 29322 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.6.234.254:44951 every 8 connection(s)
I20260620 11:46:46.037672 29255 server_base.cc:1166] Dumped server information to /tmp/dist-test-task1JRg7N/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781955746769051-7083-0/minicluster-data/master-0/data/info.pb
I20260620 11:46:46.038702 29323 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 11:46:46.038691 7083 external_mini_cluster.cc:1388] Started /tmp/dist-test-task1JRg7N/build/release/bin/kudu as pid 29255
I20260620 11:46:46.038817 7083 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task1JRg7N/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781955746769051-7083-0/minicluster-data/master-0/wal/instance
I20260620 11:46:46.038991 7083 external_mini_cluster.cc:1428] Setting key 0656e517c802b41fe987e9cadc40310b
I20260620 11:46:46.040505 29323 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.6.234.254" port: 44951 } has no permanent_uuid. Determining permanent_uuid...
WARNING: no policy specified for kudu/127.6.234.253@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.6.234.253@KRBTEST.COM" created.
Jun 20 11:46:46 dist-test-slave-cjc6 krb5kdc[29242](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781956006, etypes {rep=17 tkt=17 ses=17}, kudu/127.6.234.254@KRBTEST.COM for kudu/127.6.234.254@KRBTEST.COM
I20260620 11:46:46.052942 29327 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 11:46:46.041442 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48558 (local address 127.6.234.254:44951)
0620 11:46:46.041609 (+ 167us) server_negotiation.cc:206] Beginning negotiation
0620 11:46:46.041614 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 11:46:46.041689 (+ 75us) server_negotiation.cc:407] Connection header received
0620 11:46:46.042307 (+ 618us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 11:46:46.042309 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 11:46:46.042329 (+ 20us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 11:46:46.042433 (+ 104us) server_negotiation.cc:226] Negotiated authn=SASL
0620 11:46:46.043002 (+ 569us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 11:46:46.043807 (+ 805us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 11:46:46.044724 (+ 917us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 11:46:46.045053 (+ 329us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 11:46:46.045190 (+ 137us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 11:46:46.049604 (+ 4414us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 11:46:46.049615 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 11:46:46.049615 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 11:46:46.049627 (+ 12us) server_negotiation.cc:922] calling sasl_server_start()
0620 11:46:46.051316 (+ 1689us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 11:46:46.051733 (+ 417us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 11:46:46.051735 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 11:46:46.051736 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 11:46:46.051818 (+ 82us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 11:46:46.052045 (+ 227us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 11:46:46.052046 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 11:46:46.052046 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 11:46:46.052266 (+ 220us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 11:46:46.052359 (+ 93us) server_negotiation.cc:1035] Waiting for connection context
0620 11:46:46.052548 (+ 189us) server_negotiation.cc:299] Negotiation successful
0620 11:46:46.052693 (+ 145us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":145,"thread_start_us":84,"threads_started":1}
I20260620 11:46:46.052942 29326 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 11:46:46.041375 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.234.254:44951 (local address 127.0.0.1:48558)
0620 11:46:46.041547 (+ 172us) negotiation.cc:107] Waiting for socket to connect
0620 11:46:46.041592 (+ 45us) client_negotiation.cc:174] Beginning negotiation
0620 11:46:46.042127 (+ 535us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 11:46:46.042452 (+ 325us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 11:46:46.042454 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 11:46:46.042734 (+ 280us) client_negotiation.cc:189] Negotiated authn=SASL
0620 11:46:46.042887 (+ 153us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 11:46:46.042892 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 11:46:46.043923 (+ 1031us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 11:46:46.043923 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 11:46:46.044630 (+ 707us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 11:46:46.044632 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 11:46:46.045179 (+ 547us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 11:46:46.045180 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 11:46:46.045247 (+ 67us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 11:46:46.045852 (+ 605us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 11:46:46.045861 (+ 9us) client_negotiation.cc:656] calling sasl_client_start()
0620 11:46:46.049484 (+ 3623us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 11:46:46.051427 (+ 1943us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 11:46:46.051433 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 11:46:46.051436 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0620 11:46:46.051635 (+ 199us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 11:46:46.051888 (+ 253us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 11:46:46.051889 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 11:46:46.051889 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0620 11:46:46.051978 (+ 89us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 11:46:46.052338 (+ 360us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 11:46:46.052344 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 11:46:46.052405 (+ 61us) client_negotiation.cc:769] Sending connection context
0620 11:46:46.052516 (+ 111us) client_negotiation.cc:240] Negotiation successful
0620 11:46:46.052758 (+ 242us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":129,"mutex_wait_us":100,"thread_start_us":93,"threads_started":1}
I20260620 11:46:46.053673 29323 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.6.234.253" port: 36225 } has no permanent_uuid. Determining permanent_uuid...
I20260620 11:46:46.054109 29326 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 11:46:46.053982 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.234.253:36225 (local address 127.0.0.1:59742)
0620 11:46:46.054033 (+ 51us) negotiation.cc:107] Waiting for socket to connect
0620 11:46:46.054079 (+ 46us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.234.253:36225: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":34}
W20260620 11:46:46.054232 29272 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.6.234.253:36225: connect: Connection refused (error 111)
I20260620 11:46:46.054677 29326 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 11:46:46.054572 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.234.253:36225 (local address 127.0.0.1:59744)
0620 11:46:46.054630 (+ 58us) negotiation.cc:107] Waiting for socket to connect
0620 11:46:46.054654 (+ 24us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.234.253:36225: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":43}
W20260620 11:46:46.054783 29323 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.6.234.253:36225: Network error: Client connection negotiation failed: client connection to 127.6.234.253:36225: connect: Connection refused (error 111)
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.6.234.253 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task1JRg7N/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781955746769051-7083-0/krb5kdc/kudu_127.6.234.253.keytab.
Entry for principal kudu/127.6.234.253 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task1JRg7N/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781955746769051-7083-0/krb5kdc/kudu_127.6.234.253.keytab.
I20260620 11:46:46.064091 7083 external_mini_cluster.cc:1326] Running /tmp/dist-test-task1JRg7N/build/release/bin/kudu
/tmp/dist-test-task1JRg7N/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task1JRg7N/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781955746769051-7083-0/minicluster-data/master-1/wal
--fs_data_dirs=/tmp/dist-test-task1JRg7N/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781955746769051-7083-0/minicluster-data/master-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task1JRg7N/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781955746769051-7083-0/minicluster-data/master-1/logs
--server_dump_info_path=/tmp/dist-test-task1JRg7N/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781955746769051-7083-0/minicluster-data/master-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.6.234.253:36225
--webserver_interface=127.6.234.253
--webserver_port=0
--keytab_file=/tmp/dist-test-task1JRg7N/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781955746769051-7083-0/krb5kdc/kudu_127.6.234.253.keytab
--principal=kudu/127.6.234.253
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.6.234.212:36361
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.6.234.254:44951,127.6.234.253:36225,127.6.234.252:37231
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--enable_leader_failure_detection=false
--rpc_authentication=required
--rpc_encryption=optional with env {KRB5CCNAME=/tmp/dist-test-task1JRg7N/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781955746769051-7083-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task1JRg7N/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781955746769051-7083-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task1JRg7N/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781955746769051-7083-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260620 11:46:46.104015 29323 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.6.234.253" port: 36225 } attempt: 1
I20260620 11:46:46.104552 29326 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 11:46:46.104402 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.234.253:36225 (local address 127.0.0.1:59758)
0620 11:46:46.104481 (+ 79us) negotiation.cc:107] Waiting for socket to connect
0620 11:46:46.104522 (+ 41us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.234.253:36225: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":53}
I20260620 11:46:46.105067 29326 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 11:46:46.104931 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.234.253:36225 (local address 127.0.0.1:59774)
0620 11:46:46.105006 (+ 75us) negotiation.cc:107] Waiting for socket to connect
0620 11:46:46.105044 (+ 38us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.234.253:36225: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":56}
W20260620 11:46:46.105206 29323 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.6.234.253:36225: Network error: Client connection negotiation failed: client connection to 127.6.234.253:36225: connect: Connection refused (error 111)
W20260620 11:46:46.147212 29330 flags.cc:425] Enabled unsafe flag: --enable_leader_failure_detection=false
W20260620 11:46:46.147464 29330 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 11:46:46.147481 29330 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 11:46:46.147501 29330 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 11:46:46.148681 29330 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 11:46:46.148721 29330 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 11:46:46.148736 29330 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 11:46:46.148748 29330 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 11:46:46.148761 29330 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 11:46:46.150197 29330 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.6.234.212:36361
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--enable_leader_failure_detection=false
--fs_data_dirs=/tmp/dist-test-task1JRg7N/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781955746769051-7083-0/minicluster-data/master-1/data
--fs_wal_dir=/tmp/dist-test-task1JRg7N/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781955746769051-7083-0/minicluster-data/master-1/wal
--ipki_ca_key_size=768
--master_addresses=127.6.234.254:44951,127.6.234.253:36225,127.6.234.252:37231
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task1JRg7N/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781955746769051-7083-0/krb5kdc/kudu_127.6.234.253.keytab
--principal=kudu/127.6.234.253
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.6.234.253:36225
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task1JRg7N/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781955746769051-7083-0/minicluster-data/master-1/data/info.pb
--webserver_interface=127.6.234.253
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.29330
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task1JRg7N/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781955746769051-7083-0/minicluster-data/master-1/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 20 Jun 2026 10:32:37 UTC on ccycloud.quasar-bgyofv.root.comops.site
build id 4457173
I20260620 11:46:46.150388 29330 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 11:46:46.150557 29330 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 11:46:46.155230 29336 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 11:46:46.155234 29335 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 11:46:46.155252 29338 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 11:46:46.155507 29330 server_base.cc:1034] running on GCE node
I20260620 11:46:46.155746 29330 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 11:46:46.156001 29330 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 11:46:46.157163 29330 hybrid_clock.cc:648] HybridClock initialized: now 1781956006157145 us; error 27 us; skew 500 ppm
Jun 20 11:46:46 dist-test-slave-cjc6 krb5kdc[29242](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781956006, etypes {rep=17 tkt=17 ses=17}, kudu/127.6.234.253@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 11:46:46.159523 29330 init.cc:377] Logged in from keytab as kudu/127.6.234.253@KRBTEST.COM (short username kudu)
I20260620 11:46:46.159941 29330 webserver.cc:460] Webserver started at http://127.6.234.253:46843/ using document root <none> and password file <none>
I20260620 11:46:46.160189 29330 fs_manager.cc:359] Metadata directory not provided
I20260620 11:46:46.160234 29330 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 11:46:46.160398 29330 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 11:46:46.161481 29330 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task1JRg7N/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781955746769051-7083-0/minicluster-data/master-1/data/instance:
uuid: "dd0286cbb9e04192a97f51cd95eea21d"
format_stamp: "Formatted at 2026-06-20 11:46:46 on dist-test-slave-cjc6"
server_key: "42f80a1f9f97a827de4c99b7593ba6b4"
server_key_iv: "8d831241edd4153f189589e3b90c71e2"
server_key_version: "encryptionkey@0"
I20260620 11:46:46.161857 29330 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task1JRg7N/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781955746769051-7083-0/minicluster-data/master-1/wal/instance:
uuid: "dd0286cbb9e04192a97f51cd95eea21d"
format_stamp: "Formatted at 2026-06-20 11:46:46 on dist-test-slave-cjc6"
server_key: "42f80a1f9f97a827de4c99b7593ba6b4"
server_key_iv: "8d831241edd4153f189589e3b90c71e2"
server_key_version: "encryptionkey@0"
I20260620 11:46:46.163138 29330 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.001s
I20260620 11:46:46.163836 29345 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 11:46:46.163995 29330 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.001s
I20260620 11:46:46.164094 29330 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task1JRg7N/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781955746769051-7083-0/minicluster-data/master-1/data,/tmp/dist-test-task1JRg7N/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781955746769051-7083-0/minicluster-data/master-1/wal
uuid: "dd0286cbb9e04192a97f51cd95eea21d"
format_stamp: "Formatted at 2026-06-20 11:46:46 on dist-test-slave-cjc6"
server_key: "42f80a1f9f97a827de4c99b7593ba6b4"
server_key_iv: "8d831241edd4153f189589e3b90c71e2"
server_key_version: "encryptionkey@0"
I20260620 11:46:46.164189 29330 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task1JRg7N/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781955746769051-7083-0/minicluster-data/master-1/wal
metadata directory: /tmp/dist-test-task1JRg7N/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781955746769051-7083-0/minicluster-data/master-1/wal
1 data directories: /tmp/dist-test-task1JRg7N/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781955746769051-7083-0/minicluster-data/master-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 11:46:46.173383 29323 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.6.234.253" port: 36225 } attempt: 2
I20260620 11:46:46.173842 29326 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 11:46:46.173679 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.234.253:36225 (local address 127.0.0.1:59780)
0620 11:46:46.173776 (+ 97us) negotiation.cc:107] Waiting for socket to connect
0620 11:46:46.173816 (+ 40us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.234.253:36225: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":74}
I20260620 11:46:46.174171 29326 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 11:46:46.174078 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.234.253:36225 (local address 127.0.0.1:59788)
0620 11:46:46.174123 (+ 45us) negotiation.cc:107] Waiting for socket to connect
0620 11:46:46.174146 (+ 23us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.234.253:36225: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":30}
W20260620 11:46:46.174258 29323 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.6.234.253:36225: Network error: Client connection negotiation failed: client connection to 127.6.234.253:36225: connect: Connection refused (error 111)
I20260620 11:46:46.184131 29330 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 11:46:46.186230 29330 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 11:46:46.186395 29330 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 11:46:46.190461 29330 rpc_server.cc:307] RPC server started. Bound to: 127.6.234.253:36225
I20260620 11:46:46.190497 29397 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.6.234.253:36225 every 8 connection(s)
I20260620 11:46:46.190853 29330 server_base.cc:1166] Dumped server information to /tmp/dist-test-task1JRg7N/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781955746769051-7083-0/minicluster-data/master-1/data/info.pb
I20260620 11:46:46.191829 29398 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 11:46:46.193351 29398 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.6.234.254" port: 44951 } has no permanent_uuid. Determining permanent_uuid...
Jun 20 11:46:46 dist-test-slave-cjc6 krb5kdc[29242](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781956006, etypes {rep=17 tkt=17 ses=17}, kudu/127.6.234.253@KRBTEST.COM for kudu/127.6.234.254@KRBTEST.COM
I20260620 11:46:46.199066 7083 external_mini_cluster.cc:1388] Started /tmp/dist-test-task1JRg7N/build/release/bin/kudu as pid 29330
I20260620 11:46:46.199178 7083 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task1JRg7N/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781955746769051-7083-0/minicluster-data/master-1/wal/instance
I20260620 11:46:46.199388 7083 external_mini_cluster.cc:1428] Setting key 68d22035b5bd820df466b39d73118c9e
I20260620 11:46:46.202653 29327 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 11:46:46.194105 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48574 (local address 127.6.234.254:44951)
0620 11:46:46.194244 (+ 139us) server_negotiation.cc:206] Beginning negotiation
0620 11:46:46.194247 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0620 11:46:46.194405 (+ 158us) server_negotiation.cc:407] Connection header received
0620 11:46:46.194942 (+ 537us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 11:46:46.194943 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 11:46:46.194959 (+ 16us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 11:46:46.195023 (+ 64us) server_negotiation.cc:226] Negotiated authn=SASL
0620 11:46:46.195634 (+ 611us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 11:46:46.196122 (+ 488us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 11:46:46.197144 (+ 1022us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 11:46:46.197366 (+ 222us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 11:46:46.197448 (+ 82us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 11:46:46.199612 (+ 2164us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 11:46:46.199613 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 11:46:46.199614 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 11:46:46.199622 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0620 11:46:46.201135 (+ 1513us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 11:46:46.201623 (+ 488us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 11:46:46.201624 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 11:46:46.201624 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 11:46:46.201672 (+ 48us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 11:46:46.202027 (+ 355us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 11:46:46.202028 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 11:46:46.202028 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 11:46:46.202134 (+ 106us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 11:46:46.202186 (+ 52us) server_negotiation.cc:1035] Waiting for connection context
0620 11:46:46.202522 (+ 336us) server_negotiation.cc:299] Negotiation successful
0620 11:46:46.202589 (+ 67us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":98}
I20260620 11:46:46.202952 29400 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 11:46:46.194030 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.234.254:44951 (local address 127.0.0.1:48574)
0620 11:46:46.194266 (+ 236us) negotiation.cc:107] Waiting for socket to connect
0620 11:46:46.194317 (+ 51us) client_negotiation.cc:174] Beginning negotiation
0620 11:46:46.194826 (+ 509us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 11:46:46.195098 (+ 272us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 11:46:46.195102 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 11:46:46.195353 (+ 251us) client_negotiation.cc:189] Negotiated authn=SASL
0620 11:46:46.195537 (+ 184us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 11:46:46.195541 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 11:46:46.196241 (+ 700us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 11:46:46.196242 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 11:46:46.197050 (+ 808us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 11:46:46.197052 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 11:46:46.197473 (+ 421us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 11:46:46.197474 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 11:46:46.197593 (+ 119us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 11:46:46.198069 (+ 476us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 11:46:46.198077 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0620 11:46:46.199514 (+ 1437us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 11:46:46.201220 (+ 1706us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 11:46:46.201227 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 11:46:46.201229 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0620 11:46:46.201541 (+ 312us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 11:46:46.201814 (+ 273us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 11:46:46.201815 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 11:46:46.201815 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0620 11:46:46.201959 (+ 144us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 11:46:46.202201 (+ 242us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 11:46:46.202208 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 11:46:46.202379 (+ 171us) client_negotiation.cc:769] Sending connection context
0620 11:46:46.202493 (+ 114us) client_negotiation.cc:240] Negotiation successful
0620 11:46:46.202665 (+ 172us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":169,"thread_start_us":92,"threads_started":1}
I20260620 11:46:46.203598 29398 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.6.234.253" port: 36225 } has no permanent_uuid. Determining permanent_uuid...
WARNING: no policy specified for kudu/127.6.234.252@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.6.234.252@KRBTEST.COM" created.
Jun 20 11:46:46 dist-test-slave-cjc6 krb5kdc[29242](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781956006, etypes {rep=17 tkt=17 ses=17}, kudu/127.6.234.253@KRBTEST.COM for kudu/127.6.234.253@KRBTEST.COM
I20260620 11:46:46.214221 29400 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 11:46:46.203886 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.234.253:36225 (local address 127.0.0.1:59796)
0620 11:46:46.203946 (+ 60us) negotiation.cc:107] Waiting for socket to connect
0620 11:46:46.203958 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0620 11:46:46.203997 (+ 39us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 11:46:46.204309 (+ 312us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 11:46:46.204310 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 11:46:46.204421 (+ 111us) client_negotiation.cc:189] Negotiated authn=SASL
0620 11:46:46.204468 (+ 47us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 11:46:46.204469 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 11:46:46.205864 (+ 1395us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 11:46:46.205865 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 11:46:46.206347 (+ 482us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 11:46:46.206348 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 11:46:46.207011 (+ 663us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 11:46:46.207011 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 11:46:46.207061 (+ 50us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 11:46:46.207532 (+ 471us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 11:46:46.207535 (+ 3us) client_negotiation.cc:656] calling sasl_client_start()
0620 11:46:46.211025 (+ 3490us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 11:46:46.213093 (+ 2068us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 11:46:46.213094 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 11:46:46.213094 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0620 11:46:46.213259 (+ 165us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 11:46:46.213481 (+ 222us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 11:46:46.213482 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 11:46:46.213482 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0620 11:46:46.213535 (+ 53us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 11:46:46.213891 (+ 356us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 11:46:46.213892 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 11:46:46.213930 (+ 38us) client_negotiation.cc:769] Sending connection context
0620 11:46:46.214002 (+ 72us) client_negotiation.cc:240] Negotiation successful
0620 11:46:46.214105 (+ 103us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":31}
I20260620 11:46:46.214313 29402 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 11:46:46.203969 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59796 (local address 127.6.234.253:36225)
0620 11:46:46.204139 (+ 170us) server_negotiation.cc:206] Beginning negotiation
0620 11:46:46.204144 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 11:46:46.204170 (+ 26us) server_negotiation.cc:407] Connection header received
0620 11:46:46.204195 (+ 25us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 11:46:46.204197 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 11:46:46.204212 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 11:46:46.204301 (+ 89us) server_negotiation.cc:226] Negotiated authn=SASL
0620 11:46:46.204544 (+ 243us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 11:46:46.205791 (+ 1247us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 11:46:46.206431 (+ 640us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 11:46:46.206932 (+ 501us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 11:46:46.207016 (+ 84us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 11:46:46.211129 (+ 4113us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 11:46:46.211137 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 11:46:46.211138 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 11:46:46.211144 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0620 11:46:46.213000 (+ 1856us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 11:46:46.213351 (+ 351us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 11:46:46.213355 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 11:46:46.213355 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 11:46:46.213412 (+ 57us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 11:46:46.213607 (+ 195us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 11:46:46.213608 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 11:46:46.213609 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 11:46:46.213821 (+ 212us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 11:46:46.213914 (+ 93us) server_negotiation.cc:1035] Waiting for connection context
0620 11:46:46.214038 (+ 124us) server_negotiation.cc:299] Negotiation successful
0620 11:46:46.214173 (+ 135us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":54,"server-negotiator.queue_time_us":129,"thread_start_us":83,"threads_started":1}
I20260620 11:46:46.214880 29398 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.6.234.252" port: 37231 } has no permanent_uuid. Determining permanent_uuid...
I20260620 11:46:46.215235 29400 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 11:46:46.215116 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.234.252:37231 (local address 127.0.0.1:47930)
0620 11:46:46.215163 (+ 47us) negotiation.cc:107] Waiting for socket to connect
0620 11:46:46.215205 (+ 42us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.234.252:37231: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":30}
W20260620 11:46:46.215394 29348 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.6.234.252:37231: connect: Connection refused (error 111)
I20260620 11:46:46.215785 29400 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 11:46:46.215714 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.234.252:37231 (local address 127.0.0.1:47938)
0620 11:46:46.215742 (+ 28us) negotiation.cc:107] Waiting for socket to connect
0620 11:46:46.215765 (+ 23us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.234.252:37231: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":15}
W20260620 11:46:46.215919 29398 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.6.234.252:37231: Network error: Client connection negotiation failed: client connection to 127.6.234.252:37231: connect: Connection refused (error 111)
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.6.234.252 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task1JRg7N/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781955746769051-7083-0/krb5kdc/kudu_127.6.234.252.keytab.
Entry for principal kudu/127.6.234.252 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task1JRg7N/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781955746769051-7083-0/krb5kdc/kudu_127.6.234.252.keytab.
I20260620 11:46:46.224807 7083 external_mini_cluster.cc:1326] Running /tmp/dist-test-task1JRg7N/build/release/bin/kudu
/tmp/dist-test-task1JRg7N/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task1JRg7N/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781955746769051-7083-0/minicluster-data/master-2/wal
--fs_data_dirs=/tmp/dist-test-task1JRg7N/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781955746769051-7083-0/minicluster-data/master-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task1JRg7N/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781955746769051-7083-0/minicluster-data/master-2/logs
--server_dump_info_path=/tmp/dist-test-task1JRg7N/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781955746769051-7083-0/minicluster-data/master-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.6.234.252:37231
--webserver_interface=127.6.234.252
--webserver_port=0
--keytab_file=/tmp/dist-test-task1JRg7N/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781955746769051-7083-0/krb5kdc/kudu_127.6.234.252.keytab
--principal=kudu/127.6.234.252
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.6.234.212:36361
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.6.234.254:44951,127.6.234.253:36225,127.6.234.252:37231
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--enable_leader_failure_detection=false
--rpc_authentication=required
--rpc_encryption=optional with env {KRB5CCNAME=/tmp/dist-test-task1JRg7N/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781955746769051-7083-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task1JRg7N/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781955746769051-7083-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task1JRg7N/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781955746769051-7083-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260620 11:46:46.265244 29398 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.6.234.252" port: 37231 } attempt: 1
I20260620 11:46:46.265440 29323 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.6.234.253" port: 36225 } attempt: 3
I20260620 11:46:46.265870 29400 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 11:46:46.265732 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.234.252:37231 (local address 127.0.0.1:47942)
0620 11:46:46.265799 (+ 67us) negotiation.cc:107] Waiting for socket to connect
0620 11:46:46.265832 (+ 33us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.234.252:37231: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":44}
I20260620 11:46:46.266238 29400 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 11:46:46.266113 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.234.252:37231 (local address 127.0.0.1:47958)
0620 11:46:46.266167 (+ 54us) negotiation.cc:107] Waiting for socket to connect
0620 11:46:46.266198 (+ 31us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.234.252:37231: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":42}
W20260620 11:46:46.266394 29398 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.6.234.252:37231: Network error: Client connection negotiation failed: client connection to 127.6.234.252:37231: connect: Connection refused (error 111)
Jun 20 11:46:46 dist-test-slave-cjc6 krb5kdc[29242](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781956006, etypes {rep=17 tkt=17 ses=17}, kudu/127.6.234.254@KRBTEST.COM for kudu/127.6.234.253@KRBTEST.COM
I20260620 11:46:46.272508 29402 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 11:46:46.265805 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59802 (local address 127.6.234.253:36225)
0620 11:46:46.265851 (+ 46us) server_negotiation.cc:206] Beginning negotiation
0620 11:46:46.265853 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 11:46:46.265883 (+ 30us) server_negotiation.cc:407] Connection header received
0620 11:46:46.265984 (+ 101us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 11:46:46.265985 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 11:46:46.265998 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 11:46:46.266070 (+ 72us) server_negotiation.cc:226] Negotiated authn=SASL
0620 11:46:46.266486 (+ 416us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 11:46:46.267086 (+ 600us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 11:46:46.267818 (+ 732us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 11:46:46.268097 (+ 279us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 11:46:46.268173 (+ 76us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 11:46:46.270036 (+ 1863us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 11:46:46.270037 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 11:46:46.270038 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 11:46:46.270045 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0620 11:46:46.271397 (+ 1352us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 11:46:46.271754 (+ 357us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 11:46:46.271756 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 11:46:46.271756 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 11:46:46.271814 (+ 58us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 11:46:46.272012 (+ 198us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 11:46:46.272013 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 11:46:46.272014 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 11:46:46.272128 (+ 114us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 11:46:46.272203 (+ 75us) server_negotiation.cc:1035] Waiting for connection context
0620 11:46:46.272343 (+ 140us) server_negotiation.cc:299] Negotiation successful
0620 11:46:46.272420 (+ 77us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":27}
I20260620 11:46:46.272516 29326 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 11:46:46.265775 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.234.253:36225 (local address 127.0.0.1:59802)
0620 11:46:46.265829 (+ 54us) negotiation.cc:107] Waiting for socket to connect
0620 11:46:46.265841 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0620 11:46:46.265899 (+ 58us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 11:46:46.266084 (+ 185us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 11:46:46.266085 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 11:46:46.266290 (+ 205us) client_negotiation.cc:189] Negotiated authn=SASL
0620 11:46:46.266386 (+ 96us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 11:46:46.266389 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 11:46:46.267166 (+ 777us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 11:46:46.267167 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 11:46:46.267731 (+ 564us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 11:46:46.267733 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 11:46:46.268171 (+ 438us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 11:46:46.268172 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 11:46:46.268257 (+ 85us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 11:46:46.268747 (+ 490us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 11:46:46.268752 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0620 11:46:46.269934 (+ 1182us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 11:46:46.271492 (+ 1558us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 11:46:46.271493 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 11:46:46.271494 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0620 11:46:46.271664 (+ 170us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 11:46:46.271882 (+ 218us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 11:46:46.271883 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 11:46:46.271883 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0620 11:46:46.271948 (+ 65us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 11:46:46.272195 (+ 247us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 11:46:46.272196 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 11:46:46.272244 (+ 48us) client_negotiation.cc:769] Sending connection context
0620 11:46:46.272318 (+ 74us) client_negotiation.cc:240] Negotiation successful
0620 11:46:46.272419 (+ 101us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":34}
I20260620 11:46:46.272897 29323 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.6.234.252" port: 37231 } has no permanent_uuid. Determining permanent_uuid...
I20260620 11:46:46.273272 29326 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 11:46:46.273138 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.234.252:37231 (local address 127.0.0.1:47960)
0620 11:46:46.273202 (+ 64us) negotiation.cc:107] Waiting for socket to connect
0620 11:46:46.273241 (+ 39us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.234.252:37231: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":41}
I20260620 11:46:46.273592 29326 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 11:46:46.273490 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.234.252:37231 (local address 127.0.0.1:47976)
0620 11:46:46.273546 (+ 56us) negotiation.cc:107] Waiting for socket to connect
0620 11:46:46.273574 (+ 28us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.234.252:37231: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":44}
W20260620 11:46:46.273712 29323 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.6.234.252:37231: Network error: Client connection negotiation failed: client connection to 127.6.234.252:37231: connect: Connection refused (error 111)
I20260620 11:46:46.304888 29323 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.6.234.252" port: 37231 } attempt: 1
I20260620 11:46:46.305404 29326 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 11:46:46.305222 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.234.252:37231 (local address 127.0.0.1:47980)
0620 11:46:46.305318 (+ 96us) negotiation.cc:107] Waiting for socket to connect
0620 11:46:46.305366 (+ 48us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.234.252:37231: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":67}
I20260620 11:46:46.305827 29326 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 11:46:46.305739 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.234.252:37231 (local address 127.0.0.1:47986)
0620 11:46:46.305773 (+ 34us) negotiation.cc:107] Waiting for socket to connect
0620 11:46:46.305803 (+ 30us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.234.252:37231: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":17}
W20260620 11:46:46.305938 29323 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.6.234.252:37231: Network error: Client connection negotiation failed: client connection to 127.6.234.252:37231: connect: Connection refused (error 111)
W20260620 11:46:46.307300 29405 flags.cc:425] Enabled unsafe flag: --enable_leader_failure_detection=false
W20260620 11:46:46.307494 29405 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 11:46:46.307512 29405 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 11:46:46.307574 29405 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 11:46:46.308758 29405 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 11:46:46.308804 29405 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 11:46:46.308818 29405 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 11:46:46.308863 29405 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 11:46:46.308884 29405 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 11:46:46.310410 29405 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.6.234.212:36361
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--enable_leader_failure_detection=false
--fs_data_dirs=/tmp/dist-test-task1JRg7N/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781955746769051-7083-0/minicluster-data/master-2/data
--fs_wal_dir=/tmp/dist-test-task1JRg7N/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781955746769051-7083-0/minicluster-data/master-2/wal
--ipki_ca_key_size=768
--master_addresses=127.6.234.254:44951,127.6.234.253:36225,127.6.234.252:37231
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task1JRg7N/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781955746769051-7083-0/krb5kdc/kudu_127.6.234.252.keytab
--principal=kudu/127.6.234.252
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.6.234.252:37231
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task1JRg7N/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781955746769051-7083-0/minicluster-data/master-2/data/info.pb
--webserver_interface=127.6.234.252
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.29405
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task1JRg7N/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781955746769051-7083-0/minicluster-data/master-2/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 20 Jun 2026 10:32:37 UTC on ccycloud.quasar-bgyofv.root.comops.site
build id 4457173
I20260620 11:46:46.310669 29405 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 11:46:46.310894 29405 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 11:46:46.315505 29410 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 11:46:46.315511 29413 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 11:46:46.315513 29411 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 11:46:46.315812 29405 server_base.cc:1034] running on GCE node
I20260620 11:46:46.316064 29405 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 11:46:46.316340 29405 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 11:46:46.317481 29405 hybrid_clock.cc:648] HybridClock initialized: now 1781956006317433 us; error 61 us; skew 500 ppm
Jun 20 11:46:46 dist-test-slave-cjc6 krb5kdc[29242](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781956006, etypes {rep=17 tkt=17 ses=17}, kudu/127.6.234.252@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 11:46:46.320091 29405 init.cc:377] Logged in from keytab as kudu/127.6.234.252@KRBTEST.COM (short username kudu)
I20260620 11:46:46.320564 29405 webserver.cc:460] Webserver started at http://127.6.234.252:35277/ using document root <none> and password file <none>
I20260620 11:46:46.320787 29405 fs_manager.cc:359] Metadata directory not provided
I20260620 11:46:46.320830 29405 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 11:46:46.320971 29405 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 11:46:46.322098 29405 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task1JRg7N/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781955746769051-7083-0/minicluster-data/master-2/data/instance:
uuid: "8c33d93fc5444fa4a3bfe66e889fb516"
format_stamp: "Formatted at 2026-06-20 11:46:46 on dist-test-slave-cjc6"
server_key: "fa84a98f1d713c60b748c30a61eebb1e"
server_key_iv: "a434afd2ab9113ad9b524d9d3afb8691"
server_key_version: "encryptionkey@0"
I20260620 11:46:46.322445 29405 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task1JRg7N/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781955746769051-7083-0/minicluster-data/master-2/wal/instance:
uuid: "8c33d93fc5444fa4a3bfe66e889fb516"
format_stamp: "Formatted at 2026-06-20 11:46:46 on dist-test-slave-cjc6"
server_key: "fa84a98f1d713c60b748c30a61eebb1e"
server_key_iv: "a434afd2ab9113ad9b524d9d3afb8691"
server_key_version: "encryptionkey@0"
I20260620 11:46:46.323699 29405 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260620 11:46:46.324383 29420 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 11:46:46.324551 29405 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260620 11:46:46.324656 29405 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task1JRg7N/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781955746769051-7083-0/minicluster-data/master-2/data,/tmp/dist-test-task1JRg7N/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781955746769051-7083-0/minicluster-data/master-2/wal
uuid: "8c33d93fc5444fa4a3bfe66e889fb516"
format_stamp: "Formatted at 2026-06-20 11:46:46 on dist-test-slave-cjc6"
server_key: "fa84a98f1d713c60b748c30a61eebb1e"
server_key_iv: "a434afd2ab9113ad9b524d9d3afb8691"
server_key_version: "encryptionkey@0"
I20260620 11:46:46.324738 29405 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task1JRg7N/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781955746769051-7083-0/minicluster-data/master-2/wal
metadata directory: /tmp/dist-test-task1JRg7N/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781955746769051-7083-0/minicluster-data/master-2/wal
1 data directories: /tmp/dist-test-task1JRg7N/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781955746769051-7083-0/minicluster-data/master-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 11:46:46.334539 29398 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.6.234.252" port: 37231 } attempt: 2
I20260620 11:46:46.334956 29400 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 11:46:46.334833 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.234.252:37231 (local address 127.0.0.1:47990)
0620 11:46:46.334892 (+ 59us) negotiation.cc:107] Waiting for socket to connect
0620 11:46:46.334928 (+ 36us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.234.252:37231: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":37}
I20260620 11:46:46.335350 29400 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 11:46:46.335265 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.234.252:37231 (local address 127.0.0.1:47996)
0620 11:46:46.335310 (+ 45us) negotiation.cc:107] Waiting for socket to connect
0620 11:46:46.335334 (+ 24us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.234.252:37231: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":32}
W20260620 11:46:46.335456 29398 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.6.234.252:37231: Network error: Client connection negotiation failed: client connection to 127.6.234.252:37231: connect: Connection refused (error 111)
I20260620 11:46:46.338409 29405 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 11:46:46.340680 29405 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 11:46:46.340837 29405 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 11:46:46.345013 29405 rpc_server.cc:307] RPC server started. Bound to: 127.6.234.252:37231
I20260620 11:46:46.345060 29472 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.6.234.252:37231 every 8 connection(s)
I20260620 11:46:46.345414 29405 server_base.cc:1166] Dumped server information to /tmp/dist-test-task1JRg7N/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781955746769051-7083-0/minicluster-data/master-2/data/info.pb
I20260620 11:46:46.346408 29473 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 11:46:46.347841 29473 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.6.234.254" port: 44951 } has no permanent_uuid. Determining permanent_uuid...
I20260620 11:46:46.349398 7083 external_mini_cluster.cc:1388] Started /tmp/dist-test-task1JRg7N/build/release/bin/kudu as pid 29405
I20260620 11:46:46.349496 7083 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task1JRg7N/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781955746769051-7083-0/minicluster-data/master-2/wal/instance
I20260620 11:46:46.349741 7083 external_mini_cluster.cc:1428] Setting key d0ae83a5375b164a9d62e9204bc49134
Jun 20 11:46:46 dist-test-slave-cjc6 krb5kdc[29242](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781956006, etypes {rep=17 tkt=17 ses=17}, kudu/127.6.234.252@KRBTEST.COM for kudu/127.6.234.254@KRBTEST.COM
Jun 20 11:46:46 dist-test-slave-cjc6 krb5kdc[29242](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781956005, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.6.234.252@KRBTEST.COM
I20260620 11:46:46.357656 29327 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 11:46:46.348465 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48588 (local address 127.6.234.254:44951)
0620 11:46:46.348513 (+ 48us) server_negotiation.cc:206] Beginning negotiation
0620 11:46:46.348515 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 11:46:46.348704 (+ 189us) server_negotiation.cc:407] Connection header received
0620 11:46:46.349158 (+ 454us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 11:46:46.349159 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 11:46:46.349167 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 11:46:46.349222 (+ 55us) server_negotiation.cc:226] Negotiated authn=SASL
0620 11:46:46.350091 (+ 869us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 11:46:46.350709 (+ 618us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 11:46:46.352000 (+ 1291us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 11:46:46.352256 (+ 256us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 11:46:46.352330 (+ 74us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 11:46:46.354894 (+ 2564us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 11:46:46.354895 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 11:46:46.354895 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 11:46:46.354901 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0620 11:46:46.356296 (+ 1395us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 11:46:46.356693 (+ 397us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 11:46:46.356694 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 11:46:46.356695 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 11:46:46.356751 (+ 56us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 11:46:46.357000 (+ 249us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 11:46:46.357001 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 11:46:46.357002 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 11:46:46.357120 (+ 118us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 11:46:46.357196 (+ 76us) server_negotiation.cc:1035] Waiting for connection context
0620 11:46:46.357498 (+ 302us) server_negotiation.cc:299] Negotiation successful
0620 11:46:46.357566 (+ 68us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":19}
I20260620 11:46:46.357913 29475 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 11:46:46.348419 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.234.254:44951 (local address 127.0.0.1:48588)
0620 11:46:46.348577 (+ 158us) negotiation.cc:107] Waiting for socket to connect
0620 11:46:46.348619 (+ 42us) client_negotiation.cc:174] Beginning negotiation
0620 11:46:46.349080 (+ 461us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 11:46:46.349312 (+ 232us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 11:46:46.349316 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 11:46:46.349672 (+ 356us) client_negotiation.cc:189] Negotiated authn=SASL
0620 11:46:46.349987 (+ 315us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 11:46:46.349993 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 11:46:46.350795 (+ 802us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 11:46:46.350796 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 11:46:46.351911 (+ 1115us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 11:46:46.351913 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 11:46:46.352351 (+ 438us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 11:46:46.352352 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 11:46:46.352445 (+ 93us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 11:46:46.352969 (+ 524us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 11:46:46.352980 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0620 11:46:46.354807 (+ 1827us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 11:46:46.356382 (+ 1575us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 11:46:46.356389 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 11:46:46.356390 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0620 11:46:46.356615 (+ 225us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 11:46:46.356827 (+ 212us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 11:46:46.356828 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 11:46:46.356828 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0620 11:46:46.356931 (+ 103us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 11:46:46.357188 (+ 257us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 11:46:46.357194 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 11:46:46.357324 (+ 130us) client_negotiation.cc:769] Sending connection context
0620 11:46:46.357460 (+ 136us) client_negotiation.cc:240] Negotiation successful
0620 11:46:46.357621 (+ 161us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":121,"thread_start_us":89,"threads_started":1}
I20260620 11:46:46.358433 29473 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.6.234.253" port: 36225 } has no permanent_uuid. Determining permanent_uuid...
I20260620 11:46:46.358992 29477 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 11:46:46.350288 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48010 (local address 127.6.234.252:37231)
0620 11:46:46.350467 (+ 179us) server_negotiation.cc:206] Beginning negotiation
0620 11:46:46.350471 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 11:46:46.350492 (+ 21us) server_negotiation.cc:407] Connection header received
0620 11:46:46.350512 (+ 20us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 11:46:46.350514 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 11:46:46.350539 (+ 25us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 11:46:46.350614 (+ 75us) server_negotiation.cc:226] Negotiated authn=SASL
0620 11:46:46.351201 (+ 587us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 11:46:46.352152 (+ 951us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 11:46:46.352984 (+ 832us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 11:46:46.353424 (+ 440us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 11:46:46.353501 (+ 77us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 11:46:46.355809 (+ 2308us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 11:46:46.355816 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 11:46:46.355817 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 11:46:46.355823 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0620 11:46:46.357293 (+ 1470us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 11:46:46.357669 (+ 376us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 11:46:46.357673 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 11:46:46.357673 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 11:46:46.357777 (+ 104us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 11:46:46.358242 (+ 465us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 11:46:46.358243 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 11:46:46.358243 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 11:46:46.358368 (+ 125us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 11:46:46.358523 (+ 155us) server_negotiation.cc:1035] Waiting for connection context
0620 11:46:46.358620 (+ 97us) server_negotiation.cc:299] Negotiation successful
0620 11:46:46.358692 (+ 72us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":150,"thread_start_us":102,"threads_started":1}
Jun 20 11:46:46 dist-test-slave-cjc6 krb5kdc[29242](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781956006, etypes {rep=17 tkt=17 ses=17}, kudu/127.6.234.252@KRBTEST.COM for kudu/127.6.234.253@KRBTEST.COM
Jun 20 11:46:46 dist-test-slave-cjc6 krb5kdc[29242](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781956005, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.6.234.254@KRBTEST.COM
I20260620 11:46:46.365806 29402 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 11:46:46.358759 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59814 (local address 127.6.234.253:36225)
0620 11:46:46.358811 (+ 52us) server_negotiation.cc:206] Beginning negotiation
0620 11:46:46.358813 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 11:46:46.358822 (+ 9us) server_negotiation.cc:407] Connection header received
0620 11:46:46.358847 (+ 25us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 11:46:46.358848 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 11:46:46.358858 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 11:46:46.358914 (+ 56us) server_negotiation.cc:226] Negotiated authn=SASL
0620 11:46:46.359415 (+ 501us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 11:46:46.359995 (+ 580us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 11:46:46.360800 (+ 805us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 11:46:46.361124 (+ 324us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 11:46:46.361179 (+ 55us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 11:46:46.363055 (+ 1876us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 11:46:46.363057 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 11:46:46.363057 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 11:46:46.363064 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0620 11:46:46.364492 (+ 1428us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 11:46:46.364968 (+ 476us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 11:46:46.364969 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 11:46:46.364969 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 11:46:46.365023 (+ 54us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 11:46:46.365248 (+ 225us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 11:46:46.365249 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 11:46:46.365249 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 11:46:46.365385 (+ 136us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 11:46:46.365455 (+ 70us) server_negotiation.cc:1035] Waiting for connection context
0620 11:46:46.365612 (+ 157us) server_negotiation.cc:299] Negotiation successful
0620 11:46:46.365683 (+ 71us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":23}
I20260620 11:46:46.365803 29475 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 11:46:46.358701 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.234.253:36225 (local address 127.0.0.1:59814)
0620 11:46:46.358738 (+ 37us) negotiation.cc:107] Waiting for socket to connect
0620 11:46:46.358749 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0620 11:46:46.358787 (+ 38us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 11:46:46.358937 (+ 150us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 11:46:46.358938 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 11:46:46.359094 (+ 156us) client_negotiation.cc:189] Negotiated authn=SASL
0620 11:46:46.359153 (+ 59us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 11:46:46.359155 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 11:46:46.360067 (+ 912us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 11:46:46.360068 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 11:46:46.360726 (+ 658us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 11:46:46.360728 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 11:46:46.361193 (+ 465us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 11:46:46.361194 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 11:46:46.361262 (+ 68us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 11:46:46.361595 (+ 333us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 11:46:46.361600 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0620 11:46:46.362970 (+ 1370us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 11:46:46.364733 (+ 1763us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 11:46:46.364733 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 11:46:46.364734 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0620 11:46:46.364885 (+ 151us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 11:46:46.365110 (+ 225us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 11:46:46.365110 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 11:46:46.365111 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0620 11:46:46.365172 (+ 61us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 11:46:46.365458 (+ 286us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 11:46:46.365460 (+ 2us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 11:46:46.365502 (+ 42us) client_negotiation.cc:769] Sending connection context
0620 11:46:46.365586 (+ 84us) client_negotiation.cc:240] Negotiation successful
0620 11:46:46.365678 (+ 92us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":21}
I20260620 11:46:46.366395 29473 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.6.234.252" port: 37231 } has no permanent_uuid. Determining permanent_uuid...
I20260620 11:46:46.366942 29327 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 11:46:46.359748 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48598 (local address 127.6.234.254:44951)
0620 11:46:46.359790 (+ 42us) server_negotiation.cc:206] Beginning negotiation
0620 11:46:46.359792 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 11:46:46.359800 (+ 8us) server_negotiation.cc:407] Connection header received
0620 11:46:46.359821 (+ 21us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 11:46:46.359822 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 11:46:46.359830 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 11:46:46.359884 (+ 54us) server_negotiation.cc:226] Negotiated authn=SASL
0620 11:46:46.360464 (+ 580us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 11:46:46.360985 (+ 521us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 11:46:46.361896 (+ 911us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 11:46:46.362068 (+ 172us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 11:46:46.362150 (+ 82us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 11:46:46.364325 (+ 2175us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 11:46:46.364327 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 11:46:46.364327 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 11:46:46.364335 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0620 11:46:46.365795 (+ 1460us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 11:46:46.366164 (+ 369us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 11:46:46.366169 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 11:46:46.366170 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 11:46:46.366223 (+ 53us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 11:46:46.366412 (+ 189us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 11:46:46.366413 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 11:46:46.366413 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 11:46:46.366533 (+ 120us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 11:46:46.366698 (+ 165us) server_negotiation.cc:1035] Waiting for connection context
0620 11:46:46.366768 (+ 70us) server_negotiation.cc:299] Negotiation successful
0620 11:46:46.366843 (+ 75us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":22}
Jun 20 11:46:46 dist-test-slave-cjc6 krb5kdc[29242](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781956006, etypes {rep=17 tkt=17 ses=17}, kudu/127.6.234.252@KRBTEST.COM for kudu/127.6.234.252@KRBTEST.COM
Jun 20 11:46:46 dist-test-slave-cjc6 krb5kdc[29242](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781956005, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.6.234.253@KRBTEST.COM
I20260620 11:46:46.373564 29475 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 11:46:46.366628 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.234.252:37231 (local address 127.0.0.1:48026)
0620 11:46:46.366670 (+ 42us) negotiation.cc:107] Waiting for socket to connect
0620 11:46:46.366679 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0620 11:46:46.366722 (+ 43us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 11:46:46.366828 (+ 106us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 11:46:46.366828 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 11:46:46.366966 (+ 138us) client_negotiation.cc:189] Negotiated authn=SASL
0620 11:46:46.367002 (+ 36us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 11:46:46.367004 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 11:46:46.367826 (+ 822us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 11:46:46.367827 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 11:46:46.368547 (+ 720us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 11:46:46.368549 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 11:46:46.369002 (+ 453us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 11:46:46.369002 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 11:46:46.369052 (+ 50us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 11:46:46.369377 (+ 325us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 11:46:46.369382 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0620 11:46:46.370633 (+ 1251us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 11:46:46.372406 (+ 1773us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 11:46:46.372408 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 11:46:46.372408 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0620 11:46:46.372632 (+ 224us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 11:46:46.372846 (+ 214us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 11:46:46.372846 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 11:46:46.372847 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0620 11:46:46.372912 (+ 65us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 11:46:46.373161 (+ 249us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 11:46:46.373162 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 11:46:46.373209 (+ 47us) client_negotiation.cc:769] Sending connection context
0620 11:46:46.373279 (+ 70us) client_negotiation.cc:240] Negotiation successful
0620 11:46:46.373386 (+ 107us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":28}
I20260620 11:46:46.373574 29477 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 11:46:46.366688 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48026 (local address 127.6.234.252:37231)
0620 11:46:46.366726 (+ 38us) server_negotiation.cc:206] Beginning negotiation
0620 11:46:46.366728 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 11:46:46.366736 (+ 8us) server_negotiation.cc:407] Connection header received
0620 11:46:46.366761 (+ 25us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 11:46:46.366762 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 11:46:46.366772 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 11:46:46.366823 (+ 51us) server_negotiation.cc:226] Negotiated authn=SASL
0620 11:46:46.367072 (+ 249us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 11:46:46.367757 (+ 685us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 11:46:46.368616 (+ 859us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 11:46:46.368911 (+ 295us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 11:46:46.368986 (+ 75us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 11:46:46.370721 (+ 1735us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 11:46:46.370722 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 11:46:46.370722 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 11:46:46.370729 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0620 11:46:46.372314 (+ 1585us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 11:46:46.372709 (+ 395us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 11:46:46.372710 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 11:46:46.372710 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 11:46:46.372768 (+ 58us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 11:46:46.372991 (+ 223us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 11:46:46.372992 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 11:46:46.372993 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 11:46:46.373093 (+ 100us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 11:46:46.373144 (+ 51us) server_negotiation.cc:1035] Waiting for connection context
0620 11:46:46.373326 (+ 182us) server_negotiation.cc:299] Negotiation successful
0620 11:46:46.373449 (+ 123us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":40,"server-negotiator.queue_time_us":19}
I20260620 11:46:46.375106 29402 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 11:46:46.367584 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59820 (local address 127.6.234.253:36225)
0620 11:46:46.367636 (+ 52us) server_negotiation.cc:206] Beginning negotiation
0620 11:46:46.367638 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 11:46:46.367685 (+ 47us) server_negotiation.cc:407] Connection header received
0620 11:46:46.367742 (+ 57us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 11:46:46.367743 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 11:46:46.367751 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 11:46:46.367817 (+ 66us) server_negotiation.cc:226] Negotiated authn=SASL
0620 11:46:46.368354 (+ 537us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 11:46:46.368923 (+ 569us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 11:46:46.369846 (+ 923us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 11:46:46.370023 (+ 177us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 11:46:46.370116 (+ 93us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 11:46:46.372424 (+ 2308us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 11:46:46.372426 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 11:46:46.372426 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 11:46:46.372433 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0620 11:46:46.374126 (+ 1693us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 11:46:46.374492 (+ 366us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 11:46:46.374492 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 11:46:46.374493 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 11:46:46.374540 (+ 47us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 11:46:46.374719 (+ 179us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 11:46:46.374720 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 11:46:46.374720 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 11:46:46.374804 (+ 84us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 11:46:46.374909 (+ 105us) server_negotiation.cc:1035] Waiting for connection context
0620 11:46:46.374999 (+ 90us) server_negotiation.cc:299] Negotiation successful
0620 11:46:46.375050 (+ 51us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":18}
I20260620 11:46:46.375794 29473 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 8c33d93fc5444fa4a3bfe66e889fb516: Bootstrap starting.
I20260620 11:46:46.376571 7083 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260620 11:46:46.376875 29473 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 8c33d93fc5444fa4a3bfe66e889fb516: Neither blocks nor log segments found. Creating new log.
I20260620 11:46:46.377158 29473 log.cc:824] T 00000000000000000000000000000000 P 8c33d93fc5444fa4a3bfe66e889fb516: Log is configured to *not* fsync() on all Append() calls
I20260620 11:46:46.378031 29473 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 8c33d93fc5444fa4a3bfe66e889fb516: No bootstrap required, opened a new log
I20260620 11:46:46.377751 29297 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "00000000000000000000000000000000"
dest_uuid: "5f9b7135cde94d50aea81953f2b45385"
from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:48598
I20260620 11:46:46.379529 29473 raft_consensus.cc:348] T 00000000000000000000000000000000 P 8c33d93fc5444fa4a3bfe66e889fb516 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5f9b7135cde94d50aea81953f2b45385" member_type: VOTER last_known_addr { host: "127.6.234.254" port: 44951 } } peers { permanent_uuid: "dd0286cbb9e04192a97f51cd95eea21d" member_type: VOTER last_known_addr { host: "127.6.234.253" port: 36225 } } peers { permanent_uuid: "8c33d93fc5444fa4a3bfe66e889fb516" member_type: VOTER last_known_addr { host: "127.6.234.252" port: 37231 } }
I20260620 11:46:46.379647 29473 raft_consensus.cc:729] T 00000000000000000000000000000000 P 8c33d93fc5444fa4a3bfe66e889fb516 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8c33d93fc5444fa4a3bfe66e889fb516, State: Initialized, Role: FOLLOWER
I20260620 11:46:46.379737 29473 consensus_queue.cc:260] T 00000000000000000000000000000000 P 8c33d93fc5444fa4a3bfe66e889fb516 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5f9b7135cde94d50aea81953f2b45385" member_type: VOTER last_known_addr { host: "127.6.234.254" port: 44951 } } peers { permanent_uuid: "dd0286cbb9e04192a97f51cd95eea21d" member_type: VOTER last_known_addr { host: "127.6.234.253" port: 36225 } } peers { permanent_uuid: "8c33d93fc5444fa4a3bfe66e889fb516" member_type: VOTER last_known_addr { host: "127.6.234.252" port: 37231 } }
I20260620 11:46:46.380050 29473 sys_catalog.cc:564] T 00000000000000000000000000000000 P 8c33d93fc5444fa4a3bfe66e889fb516 [sys.catalog]: configured and running, proceeding with master startup.
I20260620 11:46:46.380046 29479 sys_catalog.cc:455] T 00000000000000000000000000000000 P 8c33d93fc5444fa4a3bfe66e889fb516 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5f9b7135cde94d50aea81953f2b45385" member_type: VOTER last_known_addr { host: "127.6.234.254" port: 44951 } } peers { permanent_uuid: "dd0286cbb9e04192a97f51cd95eea21d" member_type: VOTER last_known_addr { host: "127.6.234.253" port: 36225 } } peers { permanent_uuid: "8c33d93fc5444fa4a3bfe66e889fb516" member_type: VOTER last_known_addr { host: "127.6.234.252" port: 37231 } } }
I20260620 11:46:46.380162 29479 sys_catalog.cc:458] T 00000000000000000000000000000000 P 8c33d93fc5444fa4a3bfe66e889fb516 [sys.catalog]: This master's current role is: FOLLOWER
I20260620 11:46:46.381076 29323 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.6.234.252" port: 37231 } attempt: 2
W20260620 11:46:46.381824 29490 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 8c33d93fc5444fa4a3bfe66e889fb516: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260620 11:46:46.381901 29490 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260620 11:46:46.382254 29405 master_runner.cc:186] Error getting master registration for 127.6.234.254:44951: OK, instance_id { permanent_uuid: "5f9b7135cde94d50aea81953f2b45385" instance_seqno: 1781956006036205 } error { code: CATALOG_MANAGER_NOT_INITIALIZED status { code: SERVICE_UNAVAILABLE message: "Catalog manager is not initialized. State: Starting" } }
I20260620 11:46:46.385056 29405 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
Jun 20 11:46:46 dist-test-slave-cjc6 krb5kdc[29242](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781956006, etypes {rep=17 tkt=17 ses=17}, kudu/127.6.234.254@KRBTEST.COM for kudu/127.6.234.252@KRBTEST.COM
I20260620 11:46:46.388854 29477 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 11:46:46.381446 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48040 (local address 127.6.234.252:37231)
0620 11:46:46.381497 (+ 51us) server_negotiation.cc:206] Beginning negotiation
0620 11:46:46.381499 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 11:46:46.381507 (+ 8us) server_negotiation.cc:407] Connection header received
0620 11:46:46.381528 (+ 21us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 11:46:46.381531 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 11:46:46.381540 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 11:46:46.381598 (+ 58us) server_negotiation.cc:226] Negotiated authn=SASL
0620 11:46:46.381985 (+ 387us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 11:46:46.382678 (+ 693us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 11:46:46.383631 (+ 953us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 11:46:46.383886 (+ 255us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 11:46:46.383947 (+ 61us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 11:46:46.386124 (+ 2177us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 11:46:46.386125 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 11:46:46.386125 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 11:46:46.386132 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0620 11:46:46.387661 (+ 1529us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 11:46:46.388059 (+ 398us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 11:46:46.388064 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 11:46:46.388065 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 11:46:46.388122 (+ 57us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 11:46:46.388330 (+ 208us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 11:46:46.388331 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 11:46:46.388331 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 11:46:46.388461 (+ 130us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 11:46:46.388540 (+ 79us) server_negotiation.cc:1035] Waiting for connection context
0620 11:46:46.388692 (+ 152us) server_negotiation.cc:299] Negotiation successful
0620 11:46:46.388764 (+ 72us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":32}
I20260620 11:46:46.388854 29326 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 11:46:46.381343 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.234.252:37231 (local address 127.0.0.1:48040)
0620 11:46:46.381397 (+ 54us) negotiation.cc:107] Waiting for socket to connect
0620 11:46:46.381410 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0620 11:46:46.381457 (+ 47us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 11:46:46.381655 (+ 198us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 11:46:46.381657 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 11:46:46.381856 (+ 199us) client_negotiation.cc:189] Negotiated authn=SASL
0620 11:46:46.381902 (+ 46us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 11:46:46.381905 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 11:46:46.382772 (+ 867us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 11:46:46.382773 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 11:46:46.383538 (+ 765us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 11:46:46.383541 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 11:46:46.384000 (+ 459us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 11:46:46.384001 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 11:46:46.384051 (+ 50us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 11:46:46.384492 (+ 441us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 11:46:46.384497 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0620 11:46:46.386021 (+ 1524us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 11:46:46.387793 (+ 1772us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 11:46:46.387794 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 11:46:46.387794 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0620 11:46:46.387968 (+ 174us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 11:46:46.388190 (+ 222us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 11:46:46.388190 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 11:46:46.388191 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0620 11:46:46.388263 (+ 72us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 11:46:46.388533 (+ 270us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 11:46:46.388535 (+ 2us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 11:46:46.388579 (+ 44us) client_negotiation.cc:769] Sending connection context
0620 11:46:46.388669 (+ 90us) client_negotiation.cc:240] Negotiation successful
0620 11:46:46.388764 (+ 95us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":33}
I20260620 11:46:46.390786 29323 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 5f9b7135cde94d50aea81953f2b45385: Bootstrap starting.
I20260620 11:46:46.391507 29323 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 5f9b7135cde94d50aea81953f2b45385: Neither blocks nor log segments found. Creating new log.
I20260620 11:46:46.391813 29323 log.cc:824] T 00000000000000000000000000000000 P 5f9b7135cde94d50aea81953f2b45385: Log is configured to *not* fsync() on all Append() calls
I20260620 11:46:46.392663 29323 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 5f9b7135cde94d50aea81953f2b45385: No bootstrap required, opened a new log
I20260620 11:46:46.394171 29323 raft_consensus.cc:348] T 00000000000000000000000000000000 P 5f9b7135cde94d50aea81953f2b45385 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5f9b7135cde94d50aea81953f2b45385" member_type: VOTER last_known_addr { host: "127.6.234.254" port: 44951 } } peers { permanent_uuid: "dd0286cbb9e04192a97f51cd95eea21d" member_type: VOTER last_known_addr { host: "127.6.234.253" port: 36225 } } peers { permanent_uuid: "8c33d93fc5444fa4a3bfe66e889fb516" member_type: VOTER last_known_addr { host: "127.6.234.252" port: 37231 } }
I20260620 11:46:46.394320 29323 raft_consensus.cc:729] T 00000000000000000000000000000000 P 5f9b7135cde94d50aea81953f2b45385 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5f9b7135cde94d50aea81953f2b45385, State: Initialized, Role: FOLLOWER
I20260620 11:46:46.394395 29323 consensus_queue.cc:260] T 00000000000000000000000000000000 P 5f9b7135cde94d50aea81953f2b45385 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5f9b7135cde94d50aea81953f2b45385" member_type: VOTER last_known_addr { host: "127.6.234.254" port: 44951 } } peers { permanent_uuid: "dd0286cbb9e04192a97f51cd95eea21d" member_type: VOTER last_known_addr { host: "127.6.234.253" port: 36225 } } peers { permanent_uuid: "8c33d93fc5444fa4a3bfe66e889fb516" member_type: VOTER last_known_addr { host: "127.6.234.252" port: 37231 } }
I20260620 11:46:46.394682 29323 sys_catalog.cc:564] T 00000000000000000000000000000000 P 5f9b7135cde94d50aea81953f2b45385 [sys.catalog]: configured and running, proceeding with master startup.
I20260620 11:46:46.394748 29297 raft_consensus.cc:482] T 00000000000000000000000000000000 P 5f9b7135cde94d50aea81953f2b45385 [term 0 FOLLOWER]: Starting forced leader election (received explicit request)
I20260620 11:46:46.394860 29297 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 5f9b7135cde94d50aea81953f2b45385 [term 0 FOLLOWER]: Advancing to term 1
I20260620 11:46:46.395680 29297 raft_consensus.cc:504] T 00000000000000000000000000000000 P 5f9b7135cde94d50aea81953f2b45385 [term 1 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5f9b7135cde94d50aea81953f2b45385" member_type: VOTER last_known_addr { host: "127.6.234.254" port: 44951 } } peers { permanent_uuid: "dd0286cbb9e04192a97f51cd95eea21d" member_type: VOTER last_known_addr { host: "127.6.234.253" port: 36225 } } peers { permanent_uuid: "8c33d93fc5444fa4a3bfe66e889fb516" member_type: VOTER last_known_addr { host: "127.6.234.252" port: 37231 } }
I20260620 11:46:46.396202 29447 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "5f9b7135cde94d50aea81953f2b45385" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: true dest_uuid: "8c33d93fc5444fa4a3bfe66e889fb516"
I20260620 11:46:46.396339 29447 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 8c33d93fc5444fa4a3bfe66e889fb516 [term 0 FOLLOWER]: Advancing to term 1
W20260620 11:46:46.396744 29504 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 5f9b7135cde94d50aea81953f2b45385: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260620 11:46:46.396799 29504 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260620 11:46:46.397229 29447 raft_consensus.cc:2454] T 00000000000000000000000000000000 P 8c33d93fc5444fa4a3bfe66e889fb516 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 5f9b7135cde94d50aea81953f2b45385 in term 1.
I20260620 11:46:46.397469 29297 leader_election.cc:288] T 00000000000000000000000000000000 P 5f9b7135cde94d50aea81953f2b45385 [CANDIDATE]: Term 1 election: Requested vote from peers dd0286cbb9e04192a97f51cd95eea21d (127.6.234.253:36225), 8c33d93fc5444fa4a3bfe66e889fb516 (127.6.234.252:37231)
I20260620 11:46:46.397879 29273 leader_election.cc:302] T 00000000000000000000000000000000 P 5f9b7135cde94d50aea81953f2b45385 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 5f9b7135cde94d50aea81953f2b45385, 8c33d93fc5444fa4a3bfe66e889fb516; no voters:
I20260620 11:46:46.399097 29255 master_runner.cc:186] Error getting master registration for 127.6.234.253:36225: OK, instance_id { permanent_uuid: "dd0286cbb9e04192a97f51cd95eea21d" instance_seqno: 1781956006189247 } error { code: CATALOG_MANAGER_NOT_INITIALIZED status { code: SERVICE_UNAVAILABLE message: "Catalog manager is not initialized. State: Starting" } }
I20260620 11:46:46.401397 29255 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20260620 11:46:46.401829 29506 sys_catalog.cc:455] T 00000000000000000000000000000000 P 5f9b7135cde94d50aea81953f2b45385 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5f9b7135cde94d50aea81953f2b45385" member_type: VOTER last_known_addr { host: "127.6.234.254" port: 44951 } } peers { permanent_uuid: "dd0286cbb9e04192a97f51cd95eea21d" member_type: VOTER last_known_addr { host: "127.6.234.253" port: 36225 } } peers { permanent_uuid: "8c33d93fc5444fa4a3bfe66e889fb516" member_type: VOTER last_known_addr { host: "127.6.234.252" port: 37231 } } }
I20260620 11:46:46.401908 29506 sys_catalog.cc:458] T 00000000000000000000000000000000 P 5f9b7135cde94d50aea81953f2b45385 [sys.catalog]: This master's current role is: FOLLOWER
I20260620 11:46:46.401960 29506 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 5f9b7135cde94d50aea81953f2b45385 [term 1 FOLLOWER]: Leader election won for term 1
I20260620 11:46:46.402025 29506 raft_consensus.cc:686] T 00000000000000000000000000000000 P 5f9b7135cde94d50aea81953f2b45385 [term 1 LEADER]: Becoming Leader. State: Replica: 5f9b7135cde94d50aea81953f2b45385, State: Running, Role: LEADER
I20260620 11:46:46.402107 29506 consensus_queue.cc:237] T 00000000000000000000000000000000 P 5f9b7135cde94d50aea81953f2b45385 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5f9b7135cde94d50aea81953f2b45385" member_type: VOTER last_known_addr { host: "127.6.234.254" port: 44951 } } peers { permanent_uuid: "dd0286cbb9e04192a97f51cd95eea21d" member_type: VOTER last_known_addr { host: "127.6.234.253" port: 36225 } } peers { permanent_uuid: "8c33d93fc5444fa4a3bfe66e889fb516" member_type: VOTER last_known_addr { host: "127.6.234.252" port: 37231 } }
I20260620 11:46:46.402570 29506 sys_catalog.cc:455] T 00000000000000000000000000000000 P 5f9b7135cde94d50aea81953f2b45385 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 5f9b7135cde94d50aea81953f2b45385. Latest consensus state: current_term: 1 leader_uuid: "5f9b7135cde94d50aea81953f2b45385" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5f9b7135cde94d50aea81953f2b45385" member_type: VOTER last_known_addr { host: "127.6.234.254" port: 44951 } } peers { permanent_uuid: "dd0286cbb9e04192a97f51cd95eea21d" member_type: VOTER last_known_addr { host: "127.6.234.253" port: 36225 } } peers { permanent_uuid: "8c33d93fc5444fa4a3bfe66e889fb516" member_type: VOTER last_known_addr { host: "127.6.234.252" port: 37231 } } }
I20260620 11:46:46.402657 29506 sys_catalog.cc:458] T 00000000000000000000000000000000 P 5f9b7135cde94d50aea81953f2b45385 [sys.catalog]: This master's current role is: LEADER
I20260620 11:46:46.407824 29512 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260620 11:46:46.408053 29512 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260620 11:46:46.411865 29447 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 8c33d93fc5444fa4a3bfe66e889fb516 [term 1 FOLLOWER]: Refusing update from remote peer 5f9b7135cde94d50aea81953f2b45385: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 11:46:46.412269 29493 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 5f9b7135cde94d50aea81953f2b45385 [LEADER]: Connected to new peer: Peer: permanent_uuid: "8c33d93fc5444fa4a3bfe66e889fb516" member_type: VOTER last_known_addr { host: "127.6.234.252" port: 37231 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 11:46:46.413652 29479 sys_catalog.cc:455] T 00000000000000000000000000000000 P 8c33d93fc5444fa4a3bfe66e889fb516 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 5f9b7135cde94d50aea81953f2b45385. Latest consensus state: current_term: 1 leader_uuid: "5f9b7135cde94d50aea81953f2b45385" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5f9b7135cde94d50aea81953f2b45385" member_type: VOTER last_known_addr { host: "127.6.234.254" port: 44951 } } peers { permanent_uuid: "dd0286cbb9e04192a97f51cd95eea21d" member_type: VOTER last_known_addr { host: "127.6.234.253" port: 36225 } } peers { permanent_uuid: "8c33d93fc5444fa4a3bfe66e889fb516" member_type: VOTER last_known_addr { host: "127.6.234.252" port: 37231 } } }
I20260620 11:46:46.413807 29479 sys_catalog.cc:458] T 00000000000000000000000000000000 P 8c33d93fc5444fa4a3bfe66e889fb516 [sys.catalog]: This master's current role is: FOLLOWER
I20260620 11:46:46.418996 29493 sys_catalog.cc:455] T 00000000000000000000000000000000 P 5f9b7135cde94d50aea81953f2b45385 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "5f9b7135cde94d50aea81953f2b45385" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5f9b7135cde94d50aea81953f2b45385" member_type: VOTER last_known_addr { host: "127.6.234.254" port: 44951 } } peers { permanent_uuid: "dd0286cbb9e04192a97f51cd95eea21d" member_type: VOTER last_known_addr { host: "127.6.234.253" port: 36225 } } peers { permanent_uuid: "8c33d93fc5444fa4a3bfe66e889fb516" member_type: VOTER last_known_addr { host: "127.6.234.252" port: 37231 } } }
I20260620 11:46:46.419104 29493 sys_catalog.cc:458] T 00000000000000000000000000000000 P 5f9b7135cde94d50aea81953f2b45385 [sys.catalog]: This master's current role is: LEADER
I20260620 11:46:46.419162 29477 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 11:46:46.403409 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48048 (local address 127.6.234.252:37231)
0620 11:46:46.403447 (+ 38us) server_negotiation.cc:206] Beginning negotiation
0620 11:46:46.403448 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0620 11:46:46.409680 (+ 6232us) server_negotiation.cc:407] Connection header received
0620 11:46:46.409812 (+ 132us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 11:46:46.409813 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 11:46:46.409823 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 11:46:46.409863 (+ 40us) server_negotiation.cc:226] Negotiated authn=SASL
0620 11:46:46.411547 (+ 1684us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 11:46:46.412190 (+ 643us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 11:46:46.413216 (+ 1026us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 11:46:46.413495 (+ 279us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 11:46:46.413537 (+ 42us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 11:46:46.415230 (+ 1693us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 11:46:46.415232 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 11:46:46.415232 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 11:46:46.415240 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0620 11:46:46.417267 (+ 2027us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 11:46:46.417678 (+ 411us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 11:46:46.417680 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 11:46:46.417680 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 11:46:46.417802 (+ 122us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 11:46:46.418017 (+ 215us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 11:46:46.418018 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 11:46:46.418018 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 11:46:46.418146 (+ 128us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 11:46:46.418485 (+ 339us) server_negotiation.cc:1035] Waiting for connection context
0620 11:46:46.418738 (+ 253us) server_negotiation.cc:299] Negotiation successful
0620 11:46:46.419054 (+ 316us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":20}
I20260620 11:46:46.419521 29479 sys_catalog.cc:455] T 00000000000000000000000000000000 P 8c33d93fc5444fa4a3bfe66e889fb516 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "5f9b7135cde94d50aea81953f2b45385" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5f9b7135cde94d50aea81953f2b45385" member_type: VOTER last_known_addr { host: "127.6.234.254" port: 44951 } } peers { permanent_uuid: "dd0286cbb9e04192a97f51cd95eea21d" member_type: VOTER last_known_addr { host: "127.6.234.253" port: 36225 } } peers { permanent_uuid: "8c33d93fc5444fa4a3bfe66e889fb516" member_type: VOTER last_known_addr { host: "127.6.234.252" port: 37231 } } }
I20260620 11:46:46.419618 29479 sys_catalog.cc:458] T 00000000000000000000000000000000 P 8c33d93fc5444fa4a3bfe66e889fb516 [sys.catalog]: This master's current role is: FOLLOWER
I20260620 11:46:46.419749 29516 mvcc.cc:205] Tried to move back new op lower bound from 7298891802251898880 to 7298891802223853568. Current Snapshot: MvccSnapshot[applied={T|T < 7298891802251898880}]
I20260620 11:46:46.420048 29517 mvcc.cc:205] Tried to move back new op lower bound from 7298891802251898880 to 7298891802223853568. Current Snapshot: MvccSnapshot[applied={T|T < 7298891802251898880}]
I20260620 11:46:46.421085 29402 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 11:46:46.403227 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59832 (local address 127.6.234.253:36225)
0620 11:46:46.403270 (+ 43us) server_negotiation.cc:206] Beginning negotiation
0620 11:46:46.403272 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 11:46:46.409599 (+ 6327us) server_negotiation.cc:407] Connection header received
0620 11:46:46.409630 (+ 31us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 11:46:46.409646 (+ 16us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 11:46:46.409658 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 11:46:46.409747 (+ 89us) server_negotiation.cc:226] Negotiated authn=SASL
0620 11:46:46.412513 (+ 2766us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 11:46:46.413207 (+ 694us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 11:46:46.414825 (+ 1618us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 11:46:46.415662 (+ 837us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 11:46:46.415710 (+ 48us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 11:46:46.417997 (+ 2287us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 11:46:46.418011 (+ 14us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 11:46:46.418011 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 11:46:46.418018 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0620 11:46:46.419640 (+ 1622us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 11:46:46.420259 (+ 619us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 11:46:46.420261 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 11:46:46.420261 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 11:46:46.420319 (+ 58us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 11:46:46.420520 (+ 201us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 11:46:46.420521 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 11:46:46.420521 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 11:46:46.420646 (+ 125us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 11:46:46.420724 (+ 78us) server_negotiation.cc:1035] Waiting for connection context
0620 11:46:46.420910 (+ 186us) server_negotiation.cc:299] Negotiation successful
0620 11:46:46.420988 (+ 78us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":21}
I20260620 11:46:46.426584 29398 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.6.234.252" port: 37231 } attempt: 3
I20260620 11:46:46.426677 29327 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 11:46:46.402993 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48614 (local address 127.6.234.254:44951)
0620 11:46:46.403035 (+ 42us) server_negotiation.cc:206] Beginning negotiation
0620 11:46:46.403038 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0620 11:46:46.409388 (+ 6350us) server_negotiation.cc:407] Connection header received
0620 11:46:46.409477 (+ 89us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 11:46:46.409478 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 11:46:46.409491 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 11:46:46.409547 (+ 56us) server_negotiation.cc:226] Negotiated authn=SASL
0620 11:46:46.410821 (+ 1274us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 11:46:46.411571 (+ 750us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 11:46:46.421848 (+ 10277us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 11:46:46.422079 (+ 231us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 11:46:46.422113 (+ 34us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 11:46:46.423661 (+ 1548us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 11:46:46.423662 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 11:46:46.423663 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 11:46:46.423669 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0620 11:46:46.425270 (+ 1601us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 11:46:46.425822 (+ 552us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 11:46:46.425824 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 11:46:46.425824 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 11:46:46.425903 (+ 79us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 11:46:46.426111 (+ 208us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 11:46:46.426124 (+ 13us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 11:46:46.426125 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 11:46:46.426249 (+ 124us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 11:46:46.426350 (+ 101us) server_negotiation.cc:1035] Waiting for connection context
0620 11:46:46.426523 (+ 173us) server_negotiation.cc:299] Negotiation successful
0620 11:46:46.426598 (+ 75us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":20}
I20260620 11:46:46.427029 29512 catalog_manager.cc:1347] Generated new cluster ID: 64f36c492bd14af386489f93e95a5711
I20260620 11:46:46.427085 29512 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
Jun 20 11:46:46 dist-test-slave-cjc6 krb5kdc[29242](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781956006, etypes {rep=17 tkt=17 ses=17}, kudu/127.6.234.253@KRBTEST.COM for kudu/127.6.234.252@KRBTEST.COM
I20260620 11:46:46.434947 29400 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 11:46:46.426830 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.234.252:37231 (local address 127.0.0.1:48062)
0620 11:46:46.427019 (+ 189us) negotiation.cc:107] Waiting for socket to connect
0620 11:46:46.427029 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0620 11:46:46.427069 (+ 40us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 11:46:46.427335 (+ 266us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 11:46:46.427337 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 11:46:46.427492 (+ 155us) client_negotiation.cc:189] Negotiated authn=SASL
0620 11:46:46.427534 (+ 42us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 11:46:46.427535 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 11:46:46.428419 (+ 884us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 11:46:46.428420 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 11:46:46.429111 (+ 691us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 11:46:46.429113 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 11:46:46.429556 (+ 443us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 11:46:46.429557 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 11:46:46.429610 (+ 53us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 11:46:46.430033 (+ 423us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 11:46:46.430038 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0620 11:46:46.431496 (+ 1458us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 11:46:46.433384 (+ 1888us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 11:46:46.433385 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 11:46:46.433385 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0620 11:46:46.433606 (+ 221us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 11:46:46.433845 (+ 239us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 11:46:46.433846 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 11:46:46.433846 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0620 11:46:46.433903 (+ 57us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 11:46:46.434632 (+ 729us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 11:46:46.434633 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 11:46:46.434695 (+ 62us) client_negotiation.cc:769] Sending connection context
0620 11:46:46.434763 (+ 68us) client_negotiation.cc:240] Negotiation successful
0620 11:46:46.434855 (+ 92us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":171}
I20260620 11:46:46.434969 29477 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 11:46:46.427140 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48062 (local address 127.6.234.252:37231)
0620 11:46:46.427188 (+ 48us) server_negotiation.cc:206] Beginning negotiation
0620 11:46:46.427190 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 11:46:46.427196 (+ 6us) server_negotiation.cc:407] Connection header received
0620 11:46:46.427225 (+ 29us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 11:46:46.427226 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 11:46:46.427234 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 11:46:46.427296 (+ 62us) server_negotiation.cc:226] Negotiated authn=SASL
0620 11:46:46.427607 (+ 311us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 11:46:46.428355 (+ 748us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 11:46:46.429194 (+ 839us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 11:46:46.429497 (+ 303us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 11:46:46.429556 (+ 59us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 11:46:46.431586 (+ 2030us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 11:46:46.431588 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 11:46:46.431588 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 11:46:46.431595 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0620 11:46:46.433284 (+ 1689us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 11:46:46.433665 (+ 381us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 11:46:46.433667 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 11:46:46.433667 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 11:46:46.433761 (+ 94us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 11:46:46.433960 (+ 199us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 11:46:46.433961 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 11:46:46.433961 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 11:46:46.434096 (+ 135us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 11:46:46.434154 (+ 58us) server_negotiation.cc:1035] Waiting for connection context
0620 11:46:46.434819 (+ 665us) server_negotiation.cc:299] Negotiation successful
0620 11:46:46.434885 (+ 66us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":20}
I20260620 11:46:46.436750 29398 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P dd0286cbb9e04192a97f51cd95eea21d: Bootstrap starting.
I20260620 11:46:46.437470 29398 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P dd0286cbb9e04192a97f51cd95eea21d: Neither blocks nor log segments found. Creating new log.
I20260620 11:46:46.437808 29398 log.cc:824] T 00000000000000000000000000000000 P dd0286cbb9e04192a97f51cd95eea21d: Log is configured to *not* fsync() on all Append() calls
I20260620 11:46:46.438968 29398 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P dd0286cbb9e04192a97f51cd95eea21d: No bootstrap required, opened a new log
I20260620 11:46:46.440934 29398 raft_consensus.cc:348] T 00000000000000000000000000000000 P dd0286cbb9e04192a97f51cd95eea21d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5f9b7135cde94d50aea81953f2b45385" member_type: VOTER last_known_addr { host: "127.6.234.254" port: 44951 } } peers { permanent_uuid: "dd0286cbb9e04192a97f51cd95eea21d" member_type: VOTER last_known_addr { host: "127.6.234.253" port: 36225 } } peers { permanent_uuid: "8c33d93fc5444fa4a3bfe66e889fb516" member_type: VOTER last_known_addr { host: "127.6.234.252" port: 37231 } }
I20260620 11:46:46.441066 29398 raft_consensus.cc:729] T 00000000000000000000000000000000 P dd0286cbb9e04192a97f51cd95eea21d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: dd0286cbb9e04192a97f51cd95eea21d, State: Initialized, Role: FOLLOWER
I20260620 11:46:46.441152 29398 consensus_queue.cc:260] T 00000000000000000000000000000000 P dd0286cbb9e04192a97f51cd95eea21d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5f9b7135cde94d50aea81953f2b45385" member_type: VOTER last_known_addr { host: "127.6.234.254" port: 44951 } } peers { permanent_uuid: "dd0286cbb9e04192a97f51cd95eea21d" member_type: VOTER last_known_addr { host: "127.6.234.253" port: 36225 } } peers { permanent_uuid: "8c33d93fc5444fa4a3bfe66e889fb516" member_type: VOTER last_known_addr { host: "127.6.234.252" port: 37231 } }
I20260620 11:46:46.441489 29398 sys_catalog.cc:564] T 00000000000000000000000000000000 P dd0286cbb9e04192a97f51cd95eea21d [sys.catalog]: configured and running, proceeding with master startup.
I20260620 11:46:46.441547 29371 raft_consensus.cc:3037] T 00000000000000000000000000000000 P dd0286cbb9e04192a97f51cd95eea21d [term 0 FOLLOWER]: Advancing to term 1
I20260620 11:46:46.441843 29372 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "5f9b7135cde94d50aea81953f2b45385" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: true dest_uuid: "dd0286cbb9e04192a97f51cd95eea21d"
I20260620 11:46:46.442708 29371 raft_consensus.cc:1261] T 00000000000000000000000000000000 P dd0286cbb9e04192a97f51cd95eea21d [term 1 FOLLOWER]: Refusing update from remote peer 5f9b7135cde94d50aea81953f2b45385: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 11:46:46.443035 29506 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 5f9b7135cde94d50aea81953f2b45385 [LEADER]: Connected to new peer: Peer: permanent_uuid: "dd0286cbb9e04192a97f51cd95eea21d" member_type: VOTER last_known_addr { host: "127.6.234.253" port: 36225 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 11:46:46.443610 29372 raft_consensus.cc:2454] T 00000000000000000000000000000000 P dd0286cbb9e04192a97f51cd95eea21d [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 5f9b7135cde94d50aea81953f2b45385 in term 1.
I20260620 11:46:46.441442 29522 sys_catalog.cc:455] T 00000000000000000000000000000000 P dd0286cbb9e04192a97f51cd95eea21d [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5f9b7135cde94d50aea81953f2b45385" member_type: VOTER last_known_addr { host: "127.6.234.254" port: 44951 } } peers { permanent_uuid: "dd0286cbb9e04192a97f51cd95eea21d" member_type: VOTER last_known_addr { host: "127.6.234.253" port: 36225 } } peers { permanent_uuid: "8c33d93fc5444fa4a3bfe66e889fb516" member_type: VOTER last_known_addr { host: "127.6.234.252" port: 37231 } } }
I20260620 11:46:46.444748 29522 sys_catalog.cc:458] T 00000000000000000000000000000000 P dd0286cbb9e04192a97f51cd95eea21d [sys.catalog]: This master's current role is: FOLLOWER
I20260620 11:46:46.445508 29534 sys_catalog.cc:455] T 00000000000000000000000000000000 P dd0286cbb9e04192a97f51cd95eea21d [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "5f9b7135cde94d50aea81953f2b45385" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5f9b7135cde94d50aea81953f2b45385" member_type: VOTER last_known_addr { host: "127.6.234.254" port: 44951 } } peers { permanent_uuid: "dd0286cbb9e04192a97f51cd95eea21d" member_type: VOTER last_known_addr { host: "127.6.234.253" port: 36225 } } peers { permanent_uuid: "8c33d93fc5444fa4a3bfe66e889fb516" member_type: VOTER last_known_addr { host: "127.6.234.252" port: 37231 } } }
I20260620 11:46:46.445643 29534 sys_catalog.cc:458] T 00000000000000000000000000000000 P dd0286cbb9e04192a97f51cd95eea21d [sys.catalog]: This master's current role is: FOLLOWER
W20260620 11:46:46.446447 29533 catalog_manager.cc:1558] T 00000000000000000000000000000000 P dd0286cbb9e04192a97f51cd95eea21d: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260620 11:46:46.446547 29533 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260620 11:46:46.446779 29522 sys_catalog.cc:455] T 00000000000000000000000000000000 P dd0286cbb9e04192a97f51cd95eea21d [sys.catalog]: SysCatalogTable state changed. Reason: New leader 5f9b7135cde94d50aea81953f2b45385. Latest consensus state: current_term: 1 leader_uuid: "5f9b7135cde94d50aea81953f2b45385" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5f9b7135cde94d50aea81953f2b45385" member_type: VOTER last_known_addr { host: "127.6.234.254" port: 44951 } } peers { permanent_uuid: "dd0286cbb9e04192a97f51cd95eea21d" member_type: VOTER last_known_addr { host: "127.6.234.253" port: 36225 } } peers { permanent_uuid: "8c33d93fc5444fa4a3bfe66e889fb516" member_type: VOTER last_known_addr { host: "127.6.234.252" port: 37231 } } }
I20260620 11:46:46.446856 29522 sys_catalog.cc:458] T 00000000000000000000000000000000 P dd0286cbb9e04192a97f51cd95eea21d [sys.catalog]: This master's current role is: FOLLOWER
I20260620 11:46:46.447245 29535 mvcc.cc:205] Tried to move back new op lower bound from 7298891802251898880 to 7298891802223853568. Current Snapshot: MvccSnapshot[applied={T|T < 7298891802251898880}]
I20260620 11:46:46.463518 29506 sys_catalog.cc:455] T 00000000000000000000000000000000 P 5f9b7135cde94d50aea81953f2b45385 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "5f9b7135cde94d50aea81953f2b45385" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5f9b7135cde94d50aea81953f2b45385" member_type: VOTER last_known_addr { host: "127.6.234.254" port: 44951 } } peers { permanent_uuid: "dd0286cbb9e04192a97f51cd95eea21d" member_type: VOTER last_known_addr { host: "127.6.234.253" port: 36225 } } peers { permanent_uuid: "8c33d93fc5444fa4a3bfe66e889fb516" member_type: VOTER last_known_addr { host: "127.6.234.252" port: 37231 } } }
I20260620 11:46:46.463644 29506 sys_catalog.cc:458] T 00000000000000000000000000000000 P 5f9b7135cde94d50aea81953f2b45385 [sys.catalog]: This master's current role is: LEADER
I20260620 11:46:46.464810 29512 catalog_manager.cc:1370] Generated new certificate authority record
I20260620 11:46:46.465539 29512 catalog_manager.cc:1504] Loading token signing keys...
I20260620 11:46:46.476935 29512 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 5f9b7135cde94d50aea81953f2b45385: Generated new TSK 0
I20260620 11:46:46.477169 29512 catalog_manager.cc:1514] Initializing in-progress tserver states...
W20260620 11:46:46.506227 29477 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 11:46:46.504470 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48074 (local address 127.6.234.252:37231)
0620 11:46:46.504518 (+ 48us) server_negotiation.cc:206] Beginning negotiation
0620 11:46:46.504521 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0620 11:46:46.505498 (+ 977us) server_negotiation.cc:407] Connection header received
0620 11:46:46.505608 (+ 110us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 11:46:46.505609 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 11:46:46.505620 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 11:46:46.505652 (+ 32us) server_negotiation.cc:226] Negotiated authn=SASL
0620 11:46:46.506141 (+ 489us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:48074: BlockingRecv error: recv got EOF from 127.0.0.1:48074 (error 108)
Metrics: {"server-negotiator.queue_time_us":23}
W20260620 11:46:46.507019 29327 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 11:46:46.504869 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48628 (local address 127.6.234.254:44951)
0620 11:46:46.504910 (+ 41us) server_negotiation.cc:206] Beginning negotiation
0620 11:46:46.504912 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 11:46:46.506480 (+ 1568us) server_negotiation.cc:407] Connection header received
0620 11:46:46.506580 (+ 100us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 11:46:46.506581 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 11:46:46.506591 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 11:46:46.506622 (+ 31us) server_negotiation.cc:226] Negotiated authn=SASL
0620 11:46:46.506950 (+ 328us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:48628: BlockingRecv error: recv got EOF from 127.0.0.1:48628 (error 108)
Metrics: {"server-negotiator.queue_time_us":21}
W20260620 11:46:46.507781 29402 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 11:46:46.505200 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59844 (local address 127.6.234.253:36225)
0620 11:46:46.505243 (+ 43us) server_negotiation.cc:206] Beginning negotiation
0620 11:46:46.505245 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 11:46:46.507262 (+ 2017us) server_negotiation.cc:407] Connection header received
0620 11:46:46.507360 (+ 98us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 11:46:46.507361 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 11:46:46.507372 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 11:46:46.507402 (+ 30us) server_negotiation.cc:226] Negotiated authn=SASL
0620 11:46:46.507717 (+ 315us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:59844: BlockingRecv error: recv got EOF from 127.0.0.1:59844 (error 108)
Metrics: {"server-negotiator.queue_time_us":23}
W20260620 11:46:46.510270 29402 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 11:46:46.509448 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59848 (local address 127.6.234.253:36225)
0620 11:46:46.509503 (+ 55us) server_negotiation.cc:206] Beginning negotiation
0620 11:46:46.509505 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 11:46:46.509748 (+ 243us) server_negotiation.cc:407] Connection header received
0620 11:46:46.509860 (+ 112us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 11:46:46.509861 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 11:46:46.509878 (+ 17us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 11:46:46.509907 (+ 29us) server_negotiation.cc:226] Negotiated authn=SASL
0620 11:46:46.510227 (+ 320us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:59848: BlockingRecv error: recv got EOF from 127.0.0.1:59848 (error 108)
Metrics: {"server-negotiator.queue_time_us":29}
W20260620 11:46:46.513334 29402 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 11:46:46.512532 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59856 (local address 127.6.234.253:36225)
0620 11:46:46.512566 (+ 34us) server_negotiation.cc:206] Beginning negotiation
0620 11:46:46.512568 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 11:46:46.512810 (+ 242us) server_negotiation.cc:407] Connection header received
0620 11:46:46.512914 (+ 104us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 11:46:46.512915 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 11:46:46.512926 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 11:46:46.512955 (+ 29us) server_negotiation.cc:226] Negotiated authn=SASL
0620 11:46:46.513294 (+ 339us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:59856: BlockingRecv error: recv got EOF from 127.0.0.1:59856 (error 108)
Metrics: {"server-negotiator.queue_time_us":18}
W20260620 11:46:46.517277 29402 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 11:46:46.516497 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59864 (local address 127.6.234.253:36225)
0620 11:46:46.516531 (+ 34us) server_negotiation.cc:206] Beginning negotiation
0620 11:46:46.516533 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 11:46:46.516776 (+ 243us) server_negotiation.cc:407] Connection header received
0620 11:46:46.516875 (+ 99us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 11:46:46.516875 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 11:46:46.516885 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 11:46:46.516913 (+ 28us) server_negotiation.cc:226] Negotiated authn=SASL
0620 11:46:46.517236 (+ 323us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:59864: BlockingRecv error: recv got EOF from 127.0.0.1:59864 (error 108)
Metrics: {"server-negotiator.queue_time_us":17}
W20260620 11:46:46.523275 29402 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 11:46:46.522445 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59866 (local address 127.6.234.253:36225)
0620 11:46:46.522483 (+ 38us) server_negotiation.cc:206] Beginning negotiation
0620 11:46:46.522485 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 11:46:46.522720 (+ 235us) server_negotiation.cc:407] Connection header received
0620 11:46:46.522821 (+ 101us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 11:46:46.522822 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 11:46:46.522833 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 11:46:46.522864 (+ 31us) server_negotiation.cc:226] Negotiated authn=SASL
0620 11:46:46.523223 (+ 359us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:59866: BlockingRecv error: recv got EOF from 127.0.0.1:59866 (error 108)
Metrics: {"server-negotiator.queue_time_us":18}
W20260620 11:46:46.533468 29402 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 11:46:46.532562 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59880 (local address 127.6.234.253:36225)
0620 11:46:46.532603 (+ 41us) server_negotiation.cc:206] Beginning negotiation
0620 11:46:46.532605 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 11:46:46.532856 (+ 251us) server_negotiation.cc:407] Connection header received
0620 11:46:46.532974 (+ 118us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 11:46:46.532975 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 11:46:46.532986 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 11:46:46.533017 (+ 31us) server_negotiation.cc:226] Negotiated authn=SASL
0620 11:46:46.533415 (+ 398us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:59880: BlockingRecv error: recv got EOF from 127.0.0.1:59880 (error 108)
Metrics: {"server-negotiator.queue_time_us":19}
W20260620 11:46:46.551767 29402 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 11:46:46.550886 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59888 (local address 127.6.234.253:36225)
0620 11:46:46.550930 (+ 44us) server_negotiation.cc:206] Beginning negotiation
0620 11:46:46.550932 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 11:46:46.551191 (+ 259us) server_negotiation.cc:407] Connection header received
0620 11:46:46.551296 (+ 105us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 11:46:46.551297 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 11:46:46.551308 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 11:46:46.551337 (+ 29us) server_negotiation.cc:226] Negotiated authn=SASL
0620 11:46:46.551707 (+ 370us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:59888: BlockingRecv error: recv got EOF from 127.0.0.1:59888 (error 108)
Metrics: {"server-negotiator.queue_time_us":21}
W20260620 11:46:46.585917 29402 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 11:46:46.585104 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59896 (local address 127.6.234.253:36225)
0620 11:46:46.585154 (+ 50us) server_negotiation.cc:206] Beginning negotiation
0620 11:46:46.585156 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 11:46:46.585312 (+ 156us) server_negotiation.cc:407] Connection header received
0620 11:46:46.585383 (+ 71us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 11:46:46.585384 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 11:46:46.585396 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 11:46:46.585470 (+ 74us) server_negotiation.cc:226] Negotiated authn=SASL
0620 11:46:46.585843 (+ 373us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:59896: BlockingRecv error: recv got EOF from 127.0.0.1:59896 (error 108)
Metrics: {"server-negotiator.queue_time_us":23}
W20260620 11:46:46.651983 29402 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 11:46:46.651147 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59912 (local address 127.6.234.253:36225)
0620 11:46:46.651207 (+ 60us) server_negotiation.cc:206] Beginning negotiation
0620 11:46:46.651210 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0620 11:46:46.651354 (+ 144us) server_negotiation.cc:407] Connection header received
0620 11:46:46.651411 (+ 57us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 11:46:46.651412 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 11:46:46.651423 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 11:46:46.651471 (+ 48us) server_negotiation.cc:226] Negotiated authn=SASL
0620 11:46:46.651906 (+ 435us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:59912: BlockingRecv error: recv got EOF from 127.0.0.1:59912 (error 108)
Metrics: {"server-negotiator.queue_time_us":24}
W20260620 11:46:46.781877 29402 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 11:46:46.781102 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59922 (local address 127.6.234.253:36225)
0620 11:46:46.781177 (+ 75us) server_negotiation.cc:206] Beginning negotiation
0620 11:46:46.781179 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 11:46:46.781270 (+ 91us) server_negotiation.cc:407] Connection header received
0620 11:46:46.781368 (+ 98us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 11:46:46.781369 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 11:46:46.781380 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 11:46:46.781447 (+ 67us) server_negotiation.cc:226] Negotiated authn=SASL
0620 11:46:46.781818 (+ 371us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:59922: BlockingRecv error: recv got EOF from 127.0.0.1:59922 (error 108)
Metrics: {"server-negotiator.queue_time_us":43}
W20260620 11:46:47.039961 29402 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 11:46:47.039247 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59928 (local address 127.6.234.253:36225)
0620 11:46:47.039317 (+ 70us) server_negotiation.cc:206] Beginning negotiation
0620 11:46:47.039319 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 11:46:47.039359 (+ 40us) server_negotiation.cc:407] Connection header received
0620 11:46:47.039462 (+ 103us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 11:46:47.039463 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 11:46:47.039481 (+ 18us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 11:46:47.039547 (+ 66us) server_negotiation.cc:226] Negotiated authn=SASL
0620 11:46:47.039909 (+ 362us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:59928: BlockingRecv error: recv got EOF from 127.0.0.1:59928 (error 108)
Metrics: {"server-negotiator.queue_time_us":42}
I20260620 11:46:47.382315 29490 catalog_manager.cc:1259] Loaded cluster ID: 64f36c492bd14af386489f93e95a5711
I20260620 11:46:47.382387 29490 catalog_manager.cc:1552] T 00000000000000000000000000000000 P 8c33d93fc5444fa4a3bfe66e889fb516: loading cluster ID for follower catalog manager: success
I20260620 11:46:47.383210 29490 catalog_manager.cc:1574] T 00000000000000000000000000000000 P 8c33d93fc5444fa4a3bfe66e889fb516: acquiring CA information for follower catalog manager: success
I20260620 11:46:47.383527 29490 catalog_manager.cc:1602] T 00000000000000000000000000000000 P 8c33d93fc5444fa4a3bfe66e889fb516: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20260620 11:46:47.447027 29533 catalog_manager.cc:1259] Loaded cluster ID: 64f36c492bd14af386489f93e95a5711
I20260620 11:46:47.447129 29533 catalog_manager.cc:1552] T 00000000000000000000000000000000 P dd0286cbb9e04192a97f51cd95eea21d: loading cluster ID for follower catalog manager: success
I20260620 11:46:47.448071 29533 catalog_manager.cc:1574] T 00000000000000000000000000000000 P dd0286cbb9e04192a97f51cd95eea21d: acquiring CA information for follower catalog manager: success
I20260620 11:46:47.448397 29533 catalog_manager.cc:1602] T 00000000000000000000000000000000 P dd0286cbb9e04192a97f51cd95eea21d: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20260620 11:46:47.555687 29604 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 11:46:47.553255 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59938 (local address 127.6.234.253:36225)
0620 11:46:47.553362 (+ 107us) server_negotiation.cc:206] Beginning negotiation
0620 11:46:47.553364 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 11:46:47.553374 (+ 10us) server_negotiation.cc:407] Connection header received
0620 11:46:47.553432 (+ 58us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 11:46:47.553433 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 11:46:47.553480 (+ 47us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 11:46:47.553555 (+ 75us) server_negotiation.cc:226] Negotiated authn=TOKEN
0620 11:46:47.553754 (+ 199us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 11:46:47.554249 (+ 495us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 11:46:47.555011 (+ 762us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 11:46:47.555195 (+ 184us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 11:46:47.555244 (+ 49us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 11:46:47.555376 (+ 132us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0620 11:46:47.555481 (+ 105us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0620 11:46:47.555538 (+ 57us) server_negotiation.cc:1035] Waiting for connection context
0620 11:46:47.555573 (+ 35us) server_negotiation.cc:299] Negotiation successful
0620 11:46:47.555589 (+ 16us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":90,"thread_start_us":61,"threads_started":1}
I20260620 11:46:47.560266 29610 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 11:46:47.557735 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48090 (local address 127.6.234.252:37231)
0620 11:46:47.557865 (+ 130us) server_negotiation.cc:206] Beginning negotiation
0620 11:46:47.557867 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 11:46:47.557878 (+ 11us) server_negotiation.cc:407] Connection header received
0620 11:46:47.557945 (+ 67us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 11:46:47.557946 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 11:46:47.557990 (+ 44us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 11:46:47.558073 (+ 83us) server_negotiation.cc:226] Negotiated authn=TOKEN
0620 11:46:47.558203 (+ 130us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 11:46:47.558701 (+ 498us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 11:46:47.559457 (+ 756us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 11:46:47.559665 (+ 208us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 11:46:47.559752 (+ 87us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 11:46:47.559908 (+ 156us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0620 11:46:47.560014 (+ 106us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0620 11:46:47.560104 (+ 90us) server_negotiation.cc:1035] Waiting for connection context
0620 11:46:47.560146 (+ 42us) server_negotiation.cc:299] Negotiation successful
0620 11:46:47.560168 (+ 22us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":106,"thread_start_us":55,"threads_started":1}
I20260620 11:46:47.564738 29610 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 11:46:47.562164 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48098 (local address 127.6.234.252:37231)
0620 11:46:47.562206 (+ 42us) server_negotiation.cc:206] Beginning negotiation
0620 11:46:47.562207 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0620 11:46:47.562217 (+ 10us) server_negotiation.cc:407] Connection header received
0620 11:46:47.562304 (+ 87us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 11:46:47.562305 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 11:46:47.562316 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 11:46:47.562345 (+ 29us) server_negotiation.cc:226] Negotiated authn=TOKEN
0620 11:46:47.562501 (+ 156us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 11:46:47.562980 (+ 479us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 11:46:47.563981 (+ 1001us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 11:46:47.564262 (+ 281us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 11:46:47.564332 (+ 70us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 11:46:47.564461 (+ 129us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0620 11:46:47.564498 (+ 37us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0620 11:46:47.564558 (+ 60us) server_negotiation.cc:1035] Waiting for connection context
0620 11:46:47.564613 (+ 55us) server_negotiation.cc:299] Negotiation successful
0620 11:46:47.564633 (+ 20us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":21}
I20260620 11:46:47.565820 29616 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 11:46:47.561966 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48640 (local address 127.6.234.254:44951)
0620 11:46:47.562073 (+ 107us) server_negotiation.cc:206] Beginning negotiation
0620 11:46:47.562075 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 11:46:47.562233 (+ 158us) server_negotiation.cc:407] Connection header received
0620 11:46:47.562372 (+ 139us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 11:46:47.562374 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 11:46:47.562426 (+ 52us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 11:46:47.562508 (+ 82us) server_negotiation.cc:226] Negotiated authn=TOKEN
0620 11:46:47.563271 (+ 763us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 11:46:47.563756 (+ 485us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 11:46:47.564778 (+ 1022us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 11:46:47.565103 (+ 325us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 11:46:47.565169 (+ 66us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 11:46:47.565362 (+ 193us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0620 11:46:47.565441 (+ 79us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0620 11:46:47.565488 (+ 47us) server_negotiation.cc:1035] Waiting for connection context
0620 11:46:47.565586 (+ 98us) server_negotiation.cc:299] Negotiation successful
0620 11:46:47.565615 (+ 29us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":84,"thread_start_us":53,"threads_started":1}
I20260620 11:46:47.566124 29604 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 11:46:47.562220 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59940 (local address 127.6.234.253:36225)
0620 11:46:47.562286 (+ 66us) server_negotiation.cc:206] Beginning negotiation
0620 11:46:47.562288 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 11:46:47.562376 (+ 88us) server_negotiation.cc:407] Connection header received
0620 11:46:47.562434 (+ 58us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 11:46:47.562435 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 11:46:47.562445 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 11:46:47.562499 (+ 54us) server_negotiation.cc:226] Negotiated authn=TOKEN
0620 11:46:47.563175 (+ 676us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 11:46:47.563900 (+ 725us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 11:46:47.565277 (+ 1377us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 11:46:47.565539 (+ 262us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 11:46:47.565600 (+ 61us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 11:46:47.565779 (+ 179us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0620 11:46:47.565811 (+ 32us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0620 11:46:47.565850 (+ 39us) server_negotiation.cc:1035] Waiting for connection context
0620 11:46:47.566038 (+ 188us) server_negotiation.cc:299] Negotiation successful
0620 11:46:47.566062 (+ 24us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":40}
I20260620 11:46:47.571352 7083 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task1JRg7N/build/release/bin/kudu with pid 29255
I20260620 11:46:47.576565 7083 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task1JRg7N/build/release/bin/kudu with pid 29330
I20260620 11:46:47.581664 7083 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task1JRg7N/build/release/bin/kudu with pid 29405
2026-06-20T11:46:47Z chronyd exiting
[ OK ] ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys/0 (2825 ms)
[ RUN ] ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys/1
Loading random data
Initializing database '/tmp/dist-test-task1JRg7N/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781955746769051-7083-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 20 11:46:47 dist-test-slave-cjc6 krb5kdc[29621](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 20 11:46:47 dist-test-slave-cjc6 krb5kdc[29621](info): set up 2 sockets
Jun 20 11:46:47 dist-test-slave-cjc6 krb5kdc[29621](info): commencing operation
krb5kdc: starting...
W20260620 11:46:48.618041 7083 mini_kdc.cc:121] Time spent starting KDC: real 1.017s user 0.002s sys 0.003s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task1JRg7N/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781955746769051-7083-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task1JRg7N/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781955746769051-7083-0/krb5kdc/test-user.keytab.
Jun 20 11:46:48 dist-test-slave-cjc6 krb5kdc[29621](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781956008, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-20T11:46:48Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-20T11:46:48Z Disabled control of system clock
WARNING: no policy specified for kudu/127.6.234.254@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.6.234.254@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.6.234.254 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task1JRg7N/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781955746769051-7083-0/krb5kdc/kudu_127.6.234.254.keytab.
Entry for principal kudu/127.6.234.254 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task1JRg7N/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781955746769051-7083-0/krb5kdc/kudu_127.6.234.254.keytab.
I20260620 11:46:48.736233 7083 external_mini_cluster.cc:1326] Running /tmp/dist-test-task1JRg7N/build/release/bin/kudu
/tmp/dist-test-task1JRg7N/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task1JRg7N/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781955746769051-7083-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task1JRg7N/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781955746769051-7083-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task1JRg7N/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781955746769051-7083-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task1JRg7N/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781955746769051-7083-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.6.234.254:38981
--webserver_interface=127.6.234.254
--webserver_port=0
--keytab_file=/tmp/dist-test-task1JRg7N/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781955746769051-7083-0/krb5kdc/kudu_127.6.234.254.keytab
--principal=kudu/127.6.234.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.6.234.212:39215
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.6.234.254:38981,127.6.234.253:36053,127.6.234.252:44667
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--enable_leader_failure_detection=false
--rpc_authentication=required
--rpc_encryption=required with env {KRB5CCNAME=/tmp/dist-test-task1JRg7N/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781955746769051-7083-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task1JRg7N/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781955746769051-7083-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task1JRg7N/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781955746769051-7083-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 11:46:48.816071 29634 flags.cc:425] Enabled unsafe flag: --enable_leader_failure_detection=false
W20260620 11:46:48.816342 29634 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 11:46:48.816373 29634 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 11:46:48.816418 29634 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 11:46:48.817556 29634 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 11:46:48.817641 29634 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 11:46:48.817665 29634 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 11:46:48.817687 29634 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 11:46:48.817710 29634 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 11:46:48.819219 29634 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.6.234.212:39215
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--enable_leader_failure_detection=false
--fs_data_dirs=/tmp/dist-test-task1JRg7N/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781955746769051-7083-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task1JRg7N/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781955746769051-7083-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.6.234.254:38981,127.6.234.253:36053,127.6.234.252:44667
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task1JRg7N/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781955746769051-7083-0/krb5kdc/kudu_127.6.234.254.keytab
--principal=kudu/127.6.234.254
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.6.234.254:38981
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task1JRg7N/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781955746769051-7083-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.6.234.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.29634
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task1JRg7N/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781955746769051-7083-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 20 Jun 2026 10:32:37 UTC on ccycloud.quasar-bgyofv.root.comops.site
build id 4457173
I20260620 11:46:48.819497 29634 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 11:46:48.819738 29634 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 11:46:48.824450 29642 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 11:46:48.824410 29639 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 11:46:48.824410 29640 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 11:46:48.824709 29634 server_base.cc:1034] running on GCE node
I20260620 11:46:48.824934 29634 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 11:46:48.825228 29634 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 11:46:48.826432 29634 hybrid_clock.cc:648] HybridClock initialized: now 1781956008826417 us; error 27 us; skew 500 ppm
Jun 20 11:46:48 dist-test-slave-cjc6 krb5kdc[29621](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781956008, etypes {rep=17 tkt=17 ses=17}, kudu/127.6.234.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 11:46:48.828904 29634 init.cc:377] Logged in from keytab as kudu/127.6.234.254@KRBTEST.COM (short username kudu)
I20260620 11:46:48.829375 29634 webserver.cc:460] Webserver started at http://127.6.234.254:37379/ using document root <none> and password file <none>
I20260620 11:46:48.829613 29634 fs_manager.cc:359] Metadata directory not provided
I20260620 11:46:48.829679 29634 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 11:46:48.829867 29634 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 11:46:48.830940 29634 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task1JRg7N/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781955746769051-7083-0/minicluster-data/master-0/data/instance:
uuid: "ecbaec1ef2a8438da898ed54ae0773a2"
format_stamp: "Formatted at 2026-06-20 11:46:48 on dist-test-slave-cjc6"
server_key: "03920ba9d01dc0e75c0fd1934bbb9de4"
server_key_iv: "85b53e8a35514bd7fe6fc81a2ce55c69"
server_key_version: "encryptionkey@0"
I20260620 11:46:48.831284 29634 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task1JRg7N/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781955746769051-7083-0/minicluster-data/master-0/wal/instance:
uuid: "ecbaec1ef2a8438da898ed54ae0773a2"
format_stamp: "Formatted at 2026-06-20 11:46:48 on dist-test-slave-cjc6"
server_key: "03920ba9d01dc0e75c0fd1934bbb9de4"
server_key_iv: "85b53e8a35514bd7fe6fc81a2ce55c69"
server_key_version: "encryptionkey@0"
I20260620 11:46:48.832615 29634 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260620 11:46:48.833348 29649 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 11:46:48.833526 29634 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260620 11:46:48.833622 29634 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task1JRg7N/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781955746769051-7083-0/minicluster-data/master-0/data,/tmp/dist-test-task1JRg7N/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781955746769051-7083-0/minicluster-data/master-0/wal
uuid: "ecbaec1ef2a8438da898ed54ae0773a2"
format_stamp: "Formatted at 2026-06-20 11:46:48 on dist-test-slave-cjc6"
server_key: "03920ba9d01dc0e75c0fd1934bbb9de4"
server_key_iv: "85b53e8a35514bd7fe6fc81a2ce55c69"
server_key_version: "encryptionkey@0"
I20260620 11:46:48.833725 29634 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task1JRg7N/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781955746769051-7083-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task1JRg7N/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781955746769051-7083-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task1JRg7N/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781955746769051-7083-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 11:46:48.864178 29634 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 11:46:48.866231 29634 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 11:46:48.866400 29634 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 11:46:48.870329 29634 rpc_server.cc:307] RPC server started. Bound to: 127.6.234.254:38981
I20260620 11:46:48.870368 29701 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.6.234.254:38981 every 8 connection(s)
I20260620 11:46:48.870723 29634 server_base.cc:1166] Dumped server information to /tmp/dist-test-task1JRg7N/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781955746769051-7083-0/minicluster-data/master-0/data/info.pb
I20260620 11:46:48.870958 7083 external_mini_cluster.cc:1388] Started /tmp/dist-test-task1JRg7N/build/release/bin/kudu as pid 29634
I20260620 11:46:48.871075 7083 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task1JRg7N/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781955746769051-7083-0/minicluster-data/master-0/wal/instance
I20260620 11:46:48.871289 7083 external_mini_cluster.cc:1428] Setting key 29b82183fa37eacd7625fbb96191b7ce
I20260620 11:46:48.871803 29702 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 11:46:48.873340 29702 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.6.234.254" port: 38981 } has no permanent_uuid. Determining permanent_uuid...
WARNING: no policy specified for kudu/127.6.234.253@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.6.234.253@KRBTEST.COM" created.
Jun 20 11:46:48 dist-test-slave-cjc6 krb5kdc[29621](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781956008, etypes {rep=17 tkt=17 ses=17}, kudu/127.6.234.254@KRBTEST.COM for kudu/127.6.234.254@KRBTEST.COM
I20260620 11:46:48.887259 29706 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 11:46:48.874106 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51250 (local address 127.6.234.254:38981)
0620 11:46:48.874253 (+ 147us) server_negotiation.cc:206] Beginning negotiation
0620 11:46:48.874271 (+ 18us) server_negotiation.cc:399] Waiting for connection header
0620 11:46:48.874385 (+ 114us) server_negotiation.cc:407] Connection header received
0620 11:46:48.875182 (+ 797us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 11:46:48.875184 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 11:46:48.875209 (+ 25us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 11:46:48.875321 (+ 112us) server_negotiation.cc:226] Negotiated authn=SASL
0620 11:46:48.876075 (+ 754us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 11:46:48.876906 (+ 831us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 11:46:48.878371 (+ 1465us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 11:46:48.878716 (+ 345us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 11:46:48.878815 (+ 99us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 11:46:48.883699 (+ 4884us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 11:46:48.883705 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 11:46:48.883706 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 11:46:48.883713 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0620 11:46:48.885419 (+ 1706us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 11:46:48.885936 (+ 517us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 11:46:48.885946 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 11:46:48.885947 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 11:46:48.886119 (+ 172us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 11:46:48.886424 (+ 305us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 11:46:48.886424 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 11:46:48.886425 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 11:46:48.886653 (+ 228us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 11:46:48.886718 (+ 65us) server_negotiation.cc:1035] Waiting for connection context
0620 11:46:48.886981 (+ 263us) server_negotiation.cc:299] Negotiation successful
0620 11:46:48.887085 (+ 104us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":114,"thread_start_us":64,"threads_started":1}
I20260620 11:46:48.887398 29705 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 11:46:48.874043 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.234.254:38981 (local address 127.0.0.1:51250)
0620 11:46:48.874243 (+ 200us) negotiation.cc:107] Waiting for socket to connect
0620 11:46:48.874285 (+ 42us) client_negotiation.cc:174] Beginning negotiation
0620 11:46:48.875010 (+ 725us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 11:46:48.875351 (+ 341us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 11:46:48.875355 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 11:46:48.875727 (+ 372us) client_negotiation.cc:189] Negotiated authn=SASL
0620 11:46:48.875973 (+ 246us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 11:46:48.875980 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 11:46:48.877007 (+ 1027us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 11:46:48.877009 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 11:46:48.878262 (+ 1253us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 11:46:48.878266 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 11:46:48.878832 (+ 566us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 11:46:48.878834 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 11:46:48.878923 (+ 89us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 11:46:48.879462 (+ 539us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 11:46:48.879473 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0620 11:46:48.883597 (+ 4124us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 11:46:48.885517 (+ 1920us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 11:46:48.885523 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 11:46:48.885526 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0620 11:46:48.885834 (+ 308us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 11:46:48.886205 (+ 371us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 11:46:48.886207 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 11:46:48.886207 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0620 11:46:48.886332 (+ 125us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 11:46:48.886731 (+ 399us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 11:46:48.886738 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 11:46:48.886819 (+ 81us) client_negotiation.cc:769] Sending connection context
0620 11:46:48.886930 (+ 111us) client_negotiation.cc:240] Negotiation successful
0620 11:46:48.887198 (+ 268us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":136,"mutex_wait_us":84,"thread_start_us":87,"threads_started":1}
I20260620 11:46:48.888401 29702 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.6.234.253" port: 36053 } has no permanent_uuid. Determining permanent_uuid...
I20260620 11:46:48.888767 29705 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 11:46:48.888651 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.234.253:36053 (local address 127.0.0.1:48838)
0620 11:46:48.888703 (+ 52us) negotiation.cc:107] Waiting for socket to connect
0620 11:46:48.888746 (+ 43us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.234.253:36053: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":36}
W20260620 11:46:48.888932 29652 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.6.234.253:36053: connect: Connection refused (error 111)
I20260620 11:46:48.889384 29705 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 11:46:48.889297 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.234.253:36053 (local address 127.0.0.1:48850)
0620 11:46:48.889343 (+ 46us) negotiation.cc:107] Waiting for socket to connect
0620 11:46:48.889368 (+ 25us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.234.253:36053: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":34}
W20260620 11:46:48.889504 29702 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.6.234.253:36053: Network error: Client connection negotiation failed: client connection to 127.6.234.253:36053: connect: Connection refused (error 111)
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.6.234.253 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task1JRg7N/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781955746769051-7083-0/krb5kdc/kudu_127.6.234.253.keytab.
Entry for principal kudu/127.6.234.253 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task1JRg7N/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781955746769051-7083-0/krb5kdc/kudu_127.6.234.253.keytab.
I20260620 11:46:48.898630 7083 external_mini_cluster.cc:1326] Running /tmp/dist-test-task1JRg7N/build/release/bin/kudu
/tmp/dist-test-task1JRg7N/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task1JRg7N/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781955746769051-7083-0/minicluster-data/master-1/wal
--fs_data_dirs=/tmp/dist-test-task1JRg7N/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781955746769051-7083-0/minicluster-data/master-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task1JRg7N/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781955746769051-7083-0/minicluster-data/master-1/logs
--server_dump_info_path=/tmp/dist-test-task1JRg7N/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781955746769051-7083-0/minicluster-data/master-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.6.234.253:36053
--webserver_interface=127.6.234.253
--webserver_port=0
--keytab_file=/tmp/dist-test-task1JRg7N/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781955746769051-7083-0/krb5kdc/kudu_127.6.234.253.keytab
--principal=kudu/127.6.234.253
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.6.234.212:39215
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.6.234.254:38981,127.6.234.253:36053,127.6.234.252:44667
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--enable_leader_failure_detection=false
--rpc_authentication=required
--rpc_encryption=required with env {KRB5CCNAME=/tmp/dist-test-task1JRg7N/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781955746769051-7083-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task1JRg7N/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781955746769051-7083-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task1JRg7N/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781955746769051-7083-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260620 11:46:48.938699 29702 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.6.234.253" port: 36053 } attempt: 1
I20260620 11:46:48.939319 29705 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 11:46:48.939178 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.234.253:36053 (local address 127.0.0.1:48864)
0620 11:46:48.939241 (+ 63us) negotiation.cc:107] Waiting for socket to connect
0620 11:46:48.939290 (+ 49us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.234.253:36053: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":40}
I20260620 11:46:48.939692 29705 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 11:46:48.939609 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.234.253:36053 (local address 127.0.0.1:48868)
0620 11:46:48.939652 (+ 43us) negotiation.cc:107] Waiting for socket to connect
0620 11:46:48.939676 (+ 24us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.234.253:36053: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":32}
W20260620 11:46:48.939808 29702 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.6.234.253:36053: Network error: Client connection negotiation failed: client connection to 127.6.234.253:36053: connect: Connection refused (error 111)
W20260620 11:46:48.979583 29709 flags.cc:425] Enabled unsafe flag: --enable_leader_failure_detection=false
W20260620 11:46:48.979796 29709 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 11:46:48.979813 29709 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 11:46:48.979832 29709 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 11:46:48.980963 29709 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 11:46:48.981004 29709 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 11:46:48.981019 29709 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 11:46:48.981030 29709 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 11:46:48.981082 29709 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 11:46:48.982601 29709 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.6.234.212:39215
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--enable_leader_failure_detection=false
--fs_data_dirs=/tmp/dist-test-task1JRg7N/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781955746769051-7083-0/minicluster-data/master-1/data
--fs_wal_dir=/tmp/dist-test-task1JRg7N/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781955746769051-7083-0/minicluster-data/master-1/wal
--ipki_ca_key_size=768
--master_addresses=127.6.234.254:38981,127.6.234.253:36053,127.6.234.252:44667
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task1JRg7N/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781955746769051-7083-0/krb5kdc/kudu_127.6.234.253.keytab
--principal=kudu/127.6.234.253
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.6.234.253:36053
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task1JRg7N/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781955746769051-7083-0/minicluster-data/master-1/data/info.pb
--webserver_interface=127.6.234.253
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.29709
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task1JRg7N/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781955746769051-7083-0/minicluster-data/master-1/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 20 Jun 2026 10:32:37 UTC on ccycloud.quasar-bgyofv.root.comops.site
build id 4457173
I20260620 11:46:48.982862 29709 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 11:46:48.983084 29709 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 11:46:48.987886 29715 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 11:46:48.987896 29717 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 11:46:48.988091 29709 server_base.cc:1034] running on GCE node
W20260620 11:46:48.988152 29714 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 11:46:48.988425 29709 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 11:46:48.988665 29709 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 11:46:48.989787 29709 hybrid_clock.cc:648] HybridClock initialized: now 1781956008989758 us; error 35 us; skew 500 ppm
Jun 20 11:46:48 dist-test-slave-cjc6 krb5kdc[29621](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781956008, etypes {rep=17 tkt=17 ses=17}, kudu/127.6.234.253@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 11:46:48.992165 29709 init.cc:377] Logged in from keytab as kudu/127.6.234.253@KRBTEST.COM (short username kudu)
I20260620 11:46:48.992583 29709 webserver.cc:460] Webserver started at http://127.6.234.253:38613/ using document root <none> and password file <none>
I20260620 11:46:48.992815 29709 fs_manager.cc:359] Metadata directory not provided
I20260620 11:46:48.992882 29709 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 11:46:48.992978 29709 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 11:46:48.994053 29709 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task1JRg7N/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781955746769051-7083-0/minicluster-data/master-1/data/instance:
uuid: "02ab4448e0714124b680d996d3cd0f8a"
format_stamp: "Formatted at 2026-06-20 11:46:48 on dist-test-slave-cjc6"
server_key: "7fcff99c9f47d70eeabdfef1240236b4"
server_key_iv: "71077f08c81e594398c82d16ea41e9b8"
server_key_version: "encryptionkey@0"
I20260620 11:46:48.994359 29709 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task1JRg7N/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781955746769051-7083-0/minicluster-data/master-1/wal/instance:
uuid: "02ab4448e0714124b680d996d3cd0f8a"
format_stamp: "Formatted at 2026-06-20 11:46:48 on dist-test-slave-cjc6"
server_key: "7fcff99c9f47d70eeabdfef1240236b4"
server_key_iv: "71077f08c81e594398c82d16ea41e9b8"
server_key_version: "encryptionkey@0"
I20260620 11:46:48.995569 29709 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260620 11:46:48.996243 29724 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 11:46:48.996416 29709 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260620 11:46:48.996520 29709 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task1JRg7N/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781955746769051-7083-0/minicluster-data/master-1/data,/tmp/dist-test-task1JRg7N/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781955746769051-7083-0/minicluster-data/master-1/wal
uuid: "02ab4448e0714124b680d996d3cd0f8a"
format_stamp: "Formatted at 2026-06-20 11:46:48 on dist-test-slave-cjc6"
server_key: "7fcff99c9f47d70eeabdfef1240236b4"
server_key_iv: "71077f08c81e594398c82d16ea41e9b8"
server_key_version: "encryptionkey@0"
I20260620 11:46:48.996608 29709 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task1JRg7N/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781955746769051-7083-0/minicluster-data/master-1/wal
metadata directory: /tmp/dist-test-task1JRg7N/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781955746769051-7083-0/minicluster-data/master-1/wal
1 data directories: /tmp/dist-test-task1JRg7N/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781955746769051-7083-0/minicluster-data/master-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 11:46:49.007957 29702 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.6.234.253" port: 36053 } attempt: 2
I20260620 11:46:49.008419 29705 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 11:46:49.008286 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.234.253:36053 (local address 127.0.0.1:48880)
0620 11:46:49.008345 (+ 59us) negotiation.cc:107] Waiting for socket to connect
0620 11:46:49.008387 (+ 42us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.234.253:36053: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":36}
I20260620 11:46:49.008723 29705 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 11:46:49.008656 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.234.253:36053 (local address 127.0.0.1:48896)
0620 11:46:49.008682 (+ 26us) negotiation.cc:107] Waiting for socket to connect
0620 11:46:49.008705 (+ 23us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.234.253:36053: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":15}
W20260620 11:46:49.008848 29702 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.6.234.253:36053: Network error: Client connection negotiation failed: client connection to 127.6.234.253:36053: connect: Connection refused (error 111)
I20260620 11:46:49.029579 29709 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 11:46:49.031565 29709 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 11:46:49.031709 29709 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 11:46:49.035712 29709 rpc_server.cc:307] RPC server started. Bound to: 127.6.234.253:36053
I20260620 11:46:49.035769 29776 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.6.234.253:36053 every 8 connection(s)
I20260620 11:46:49.036101 29709 server_base.cc:1166] Dumped server information to /tmp/dist-test-task1JRg7N/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781955746769051-7083-0/minicluster-data/master-1/data/info.pb
I20260620 11:46:49.037207 29777 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 11:46:49.038795 29777 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.6.234.254" port: 38981 } has no permanent_uuid. Determining permanent_uuid...
I20260620 11:46:49.043617 7083 external_mini_cluster.cc:1388] Started /tmp/dist-test-task1JRg7N/build/release/bin/kudu as pid 29709
I20260620 11:46:49.043743 7083 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task1JRg7N/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781955746769051-7083-0/minicluster-data/master-1/wal/instance
I20260620 11:46:49.043957 7083 external_mini_cluster.cc:1428] Setting key 55e5d3b6b56dfd24c097d4db0e281c9e
Jun 20 11:46:49 dist-test-slave-cjc6 krb5kdc[29621](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781956008, etypes {rep=17 tkt=17 ses=17}, kudu/127.6.234.253@KRBTEST.COM for kudu/127.6.234.254@KRBTEST.COM
I20260620 11:46:49.048189 29706 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 11:46:49.039473 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51260 (local address 127.6.234.254:38981)
0620 11:46:49.039532 (+ 59us) server_negotiation.cc:206] Beginning negotiation
0620 11:46:49.039534 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 11:46:49.039699 (+ 165us) server_negotiation.cc:407] Connection header received
0620 11:46:49.040126 (+ 427us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 11:46:49.040127 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 11:46:49.040140 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 11:46:49.040202 (+ 62us) server_negotiation.cc:226] Negotiated authn=SASL
0620 11:46:49.040775 (+ 573us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 11:46:49.041266 (+ 491us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 11:46:49.042250 (+ 984us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 11:46:49.042453 (+ 203us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 11:46:49.042513 (+ 60us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 11:46:49.044948 (+ 2435us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 11:46:49.044950 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 11:46:49.044950 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 11:46:49.044957 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0620 11:46:49.046699 (+ 1742us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 11:46:49.047196 (+ 497us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 11:46:49.047197 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 11:46:49.047197 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 11:46:49.047254 (+ 57us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 11:46:49.047533 (+ 279us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 11:46:49.047533 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 11:46:49.047534 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 11:46:49.047634 (+ 100us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 11:46:49.047682 (+ 48us) server_negotiation.cc:1035] Waiting for connection context
0620 11:46:49.048036 (+ 354us) server_negotiation.cc:299] Negotiation successful
0620 11:46:49.048120 (+ 84us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":36}
I20260620 11:46:49.048436 29779 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 11:46:49.039415 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.234.254:38981 (local address 127.0.0.1:51260)
0620 11:46:49.039571 (+ 156us) negotiation.cc:107] Waiting for socket to connect
0620 11:46:49.039611 (+ 40us) client_negotiation.cc:174] Beginning negotiation
0620 11:46:49.040030 (+ 419us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 11:46:49.040282 (+ 252us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 11:46:49.040286 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 11:46:49.040519 (+ 233us) client_negotiation.cc:189] Negotiated authn=SASL
0620 11:46:49.040701 (+ 182us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 11:46:49.040704 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 11:46:49.041341 (+ 637us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 11:46:49.041341 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 11:46:49.042181 (+ 840us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 11:46:49.042184 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 11:46:49.042547 (+ 363us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 11:46:49.042548 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 11:46:49.042640 (+ 92us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 11:46:49.043102 (+ 462us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 11:46:49.043110 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0620 11:46:49.044848 (+ 1738us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 11:46:49.046790 (+ 1942us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 11:46:49.046797 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 11:46:49.046799 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0620 11:46:49.047118 (+ 319us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 11:46:49.047322 (+ 204us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 11:46:49.047322 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 11:46:49.047323 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0620 11:46:49.047469 (+ 146us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 11:46:49.047701 (+ 232us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 11:46:49.047708 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 11:46:49.047886 (+ 178us) client_negotiation.cc:769] Sending connection context
0620 11:46:49.048013 (+ 127us) client_negotiation.cc:240] Negotiation successful
0620 11:46:49.048196 (+ 183us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":116,"thread_start_us":87,"threads_started":1}
I20260620 11:46:49.049084 29777 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.6.234.253" port: 36053 } has no permanent_uuid. Determining permanent_uuid...
WARNING: no policy specified for kudu/127.6.234.252@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.6.234.252@KRBTEST.COM" created.
Jun 20 11:46:49 dist-test-slave-cjc6 krb5kdc[29621](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781956008, etypes {rep=17 tkt=17 ses=17}, kudu/127.6.234.253@KRBTEST.COM for kudu/127.6.234.253@KRBTEST.COM
I20260620 11:46:49.058810 29779 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 11:46:49.049356 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.234.253:36053 (local address 127.0.0.1:48898)
0620 11:46:49.049410 (+ 54us) negotiation.cc:107] Waiting for socket to connect
0620 11:46:49.049421 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0620 11:46:49.049460 (+ 39us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 11:46:49.049787 (+ 327us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 11:46:49.049788 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 11:46:49.049952 (+ 164us) client_negotiation.cc:189] Negotiated authn=SASL
0620 11:46:49.050016 (+ 64us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 11:46:49.050018 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 11:46:49.050843 (+ 825us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 11:46:49.050844 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 11:46:49.051683 (+ 839us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 11:46:49.051686 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 11:46:49.052122 (+ 436us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 11:46:49.052123 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 11:46:49.052208 (+ 85us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 11:46:49.052555 (+ 347us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 11:46:49.052560 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0620 11:46:49.056030 (+ 3470us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 11:46:49.057759 (+ 1729us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 11:46:49.057760 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 11:46:49.057760 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0620 11:46:49.057917 (+ 157us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 11:46:49.058133 (+ 216us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 11:46:49.058133 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 11:46:49.058134 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0620 11:46:49.058191 (+ 57us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 11:46:49.058470 (+ 279us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 11:46:49.058471 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 11:46:49.058509 (+ 38us) client_negotiation.cc:769] Sending connection context
0620 11:46:49.058593 (+ 84us) client_negotiation.cc:240] Negotiation successful
0620 11:46:49.058689 (+ 96us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":36}
I20260620 11:46:49.058869 29781 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 11:46:49.049426 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48898 (local address 127.6.234.253:36053)
0620 11:46:49.049576 (+ 150us) server_negotiation.cc:206] Beginning negotiation
0620 11:46:49.049580 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 11:46:49.049598 (+ 18us) server_negotiation.cc:407] Connection header received
0620 11:46:49.049640 (+ 42us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 11:46:49.049641 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 11:46:49.049652 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 11:46:49.049775 (+ 123us) server_negotiation.cc:226] Negotiated authn=SASL
0620 11:46:49.050101 (+ 326us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 11:46:49.050757 (+ 656us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 11:46:49.051764 (+ 1007us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 11:46:49.052042 (+ 278us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 11:46:49.052095 (+ 53us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 11:46:49.056116 (+ 4021us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 11:46:49.056124 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 11:46:49.056124 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 11:46:49.056130 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0620 11:46:49.057651 (+ 1521us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 11:46:49.058014 (+ 363us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 11:46:49.058021 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 11:46:49.058022 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 11:46:49.058068 (+ 46us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 11:46:49.058258 (+ 190us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 11:46:49.058259 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 11:46:49.058259 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 11:46:49.058393 (+ 134us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 11:46:49.058479 (+ 86us) server_negotiation.cc:1035] Waiting for connection context
0620 11:46:49.058625 (+ 146us) server_negotiation.cc:299] Negotiation successful
0620 11:46:49.058742 (+ 117us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":51,"server-negotiator.queue_time_us":123,"thread_start_us":97,"threads_started":1}
I20260620 11:46:49.059469 29777 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.6.234.252" port: 44667 } has no permanent_uuid. Determining permanent_uuid...
I20260620 11:46:49.059820 29779 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 11:46:49.059709 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.234.252:44667 (local address 127.0.0.1:59416)
0620 11:46:49.059761 (+ 52us) negotiation.cc:107] Waiting for socket to connect
0620 11:46:49.059801 (+ 40us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.234.252:44667: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":36}
W20260620 11:46:49.059983 29725 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.6.234.252:44667: connect: Connection refused (error 111)
I20260620 11:46:49.060437 29779 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 11:46:49.060356 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.234.252:44667 (local address 127.0.0.1:59432)
0620 11:46:49.060400 (+ 44us) negotiation.cc:107] Waiting for socket to connect
0620 11:46:49.060423 (+ 23us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.234.252:44667: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":32}
W20260620 11:46:49.060556 29777 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.6.234.252:44667: Network error: Client connection negotiation failed: client connection to 127.6.234.252:44667: connect: Connection refused (error 111)
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.6.234.252 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task1JRg7N/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781955746769051-7083-0/krb5kdc/kudu_127.6.234.252.keytab.
Entry for principal kudu/127.6.234.252 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task1JRg7N/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781955746769051-7083-0/krb5kdc/kudu_127.6.234.252.keytab.
I20260620 11:46:49.070680 7083 external_mini_cluster.cc:1326] Running /tmp/dist-test-task1JRg7N/build/release/bin/kudu
/tmp/dist-test-task1JRg7N/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task1JRg7N/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781955746769051-7083-0/minicluster-data/master-2/wal
--fs_data_dirs=/tmp/dist-test-task1JRg7N/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781955746769051-7083-0/minicluster-data/master-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task1JRg7N/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781955746769051-7083-0/minicluster-data/master-2/logs
--server_dump_info_path=/tmp/dist-test-task1JRg7N/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781955746769051-7083-0/minicluster-data/master-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.6.234.252:44667
--webserver_interface=127.6.234.252
--webserver_port=0
--keytab_file=/tmp/dist-test-task1JRg7N/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781955746769051-7083-0/krb5kdc/kudu_127.6.234.252.keytab
--principal=kudu/127.6.234.252
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.6.234.212:39215
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.6.234.254:38981,127.6.234.253:36053,127.6.234.252:44667
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--enable_leader_failure_detection=false
--rpc_authentication=required
--rpc_encryption=required with env {KRB5CCNAME=/tmp/dist-test-task1JRg7N/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781955746769051-7083-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task1JRg7N/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781955746769051-7083-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task1JRg7N/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781955746769051-7083-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260620 11:46:49.100028 29702 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.6.234.253" port: 36053 } attempt: 3
Jun 20 11:46:49 dist-test-slave-cjc6 krb5kdc[29621](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781956008, etypes {rep=17 tkt=17 ses=17}, kudu/127.6.234.254@KRBTEST.COM for kudu/127.6.234.253@KRBTEST.COM
I20260620 11:46:49.108683 29705 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 11:46:49.100416 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.234.253:36053 (local address 127.0.0.1:48904)
0620 11:46:49.100486 (+ 70us) negotiation.cc:107] Waiting for socket to connect
0620 11:46:49.100498 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0620 11:46:49.100560 (+ 62us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 11:46:49.100869 (+ 309us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 11:46:49.100870 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 11:46:49.101075 (+ 205us) client_negotiation.cc:189] Negotiated authn=SASL
0620 11:46:49.101169 (+ 94us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 11:46:49.101172 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 11:46:49.102315 (+ 1143us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 11:46:49.102316 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 11:46:49.103112 (+ 796us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 11:46:49.103115 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 11:46:49.103674 (+ 559us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 11:46:49.103675 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 11:46:49.103746 (+ 71us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 11:46:49.104213 (+ 467us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 11:46:49.104218 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0620 11:46:49.105906 (+ 1688us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 11:46:49.107694 (+ 1788us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 11:46:49.107695 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 11:46:49.107695 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0620 11:46:49.107894 (+ 199us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 11:46:49.108080 (+ 186us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 11:46:49.108080 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 11:46:49.108081 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0620 11:46:49.108129 (+ 48us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 11:46:49.108394 (+ 265us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 11:46:49.108395 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 11:46:49.108439 (+ 44us) client_negotiation.cc:769] Sending connection context
0620 11:46:49.108496 (+ 57us) client_negotiation.cc:240] Negotiation successful
0620 11:46:49.108586 (+ 90us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":21}
I20260620 11:46:49.109161 29781 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 11:46:49.100660 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48904 (local address 127.6.234.253:36053)
0620 11:46:49.100700 (+ 40us) server_negotiation.cc:206] Beginning negotiation
0620 11:46:49.100702 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 11:46:49.100710 (+ 8us) server_negotiation.cc:407] Connection header received
0620 11:46:49.100761 (+ 51us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 11:46:49.100761 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 11:46:49.100773 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 11:46:49.100830 (+ 57us) server_negotiation.cc:226] Negotiated authn=SASL
0620 11:46:49.101298 (+ 468us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 11:46:49.102212 (+ 914us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 11:46:49.103219 (+ 1007us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 11:46:49.103556 (+ 337us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 11:46:49.103615 (+ 59us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 11:46:49.106006 (+ 2391us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 11:46:49.106008 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 11:46:49.106009 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 11:46:49.106017 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0620 11:46:49.107605 (+ 1588us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 11:46:49.107966 (+ 361us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 11:46:49.107967 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 11:46:49.107968 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 11:46:49.108019 (+ 51us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 11:46:49.108186 (+ 167us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 11:46:49.108186 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 11:46:49.108187 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 11:46:49.108317 (+ 130us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 11:46:49.108370 (+ 53us) server_negotiation.cc:1035] Waiting for connection context
0620 11:46:49.108994 (+ 624us) server_negotiation.cc:299] Negotiation successful
0620 11:46:49.109071 (+ 77us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":19}
I20260620 11:46:49.109771 29777 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.6.234.252" port: 44667 } attempt: 1
I20260620 11:46:49.113656 29779 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 11:46:49.113553 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.234.252:44667 (local address 127.0.0.1:59436)
0620 11:46:49.113589 (+ 36us) negotiation.cc:107] Waiting for socket to connect
0620 11:46:49.113618 (+ 29us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.234.252:44667: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":19}
I20260620 11:46:49.113793 29702 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.6.234.252" port: 44667 } has no permanent_uuid. Determining permanent_uuid...
I20260620 11:46:49.114107 29705 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 11:46:49.114012 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.234.252:44667 (local address 127.0.0.1:59446)
0620 11:46:49.114052 (+ 40us) negotiation.cc:107] Waiting for socket to connect
0620 11:46:49.114076 (+ 24us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.234.252:44667: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":25}
I20260620 11:46:49.114154 29779 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 11:46:49.114076 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.234.252:44667 (local address 127.0.0.1:59454)
0620 11:46:49.114109 (+ 33us) negotiation.cc:107] Waiting for socket to connect
0620 11:46:49.114129 (+ 20us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.234.252:44667: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":22}
W20260620 11:46:49.114234 29777 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.6.234.252:44667: Network error: Client connection negotiation failed: client connection to 127.6.234.252:44667: connect: Connection refused (error 111)
I20260620 11:46:49.114603 29705 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 11:46:49.114474 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.234.252:44667 (local address 127.0.0.1:59458)
0620 11:46:49.114518 (+ 44us) negotiation.cc:107] Waiting for socket to connect
0620 11:46:49.114575 (+ 57us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.234.252:44667: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":32}
W20260620 11:46:49.114679 29702 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.6.234.252:44667: Network error: Client connection negotiation failed: client connection to 127.6.234.252:44667: connect: Connection refused (error 111)
I20260620 11:46:49.145846 29702 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.6.234.252" port: 44667 } attempt: 1
I20260620 11:46:49.146313 29705 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 11:46:49.146160 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.234.252:44667 (local address 127.0.0.1:59466)
0620 11:46:49.146228 (+ 68us) negotiation.cc:107] Waiting for socket to connect
0620 11:46:49.146277 (+ 49us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.234.252:44667: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":39}
I20260620 11:46:49.146709 29705 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 11:46:49.146604 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.234.252:44667 (local address 127.0.0.1:59482)
0620 11:46:49.146655 (+ 51us) negotiation.cc:107] Waiting for socket to connect
0620 11:46:49.146686 (+ 31us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.234.252:44667: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":35}
W20260620 11:46:49.146831 29702 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.6.234.252:44667: Network error: Client connection negotiation failed: client connection to 127.6.234.252:44667: connect: Connection refused (error 111)
W20260620 11:46:49.157871 29784 flags.cc:425] Enabled unsafe flag: --enable_leader_failure_detection=false
W20260620 11:46:49.158072 29784 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 11:46:49.158115 29784 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 11:46:49.158154 29784 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 11:46:49.159318 29784 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 11:46:49.159358 29784 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 11:46:49.159372 29784 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 11:46:49.159420 29784 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 11:46:49.159443 29784 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 11:46:49.160971 29784 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.6.234.212:39215
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--enable_leader_failure_detection=false
--fs_data_dirs=/tmp/dist-test-task1JRg7N/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781955746769051-7083-0/minicluster-data/master-2/data
--fs_wal_dir=/tmp/dist-test-task1JRg7N/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781955746769051-7083-0/minicluster-data/master-2/wal
--ipki_ca_key_size=768
--master_addresses=127.6.234.254:38981,127.6.234.253:36053,127.6.234.252:44667
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task1JRg7N/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781955746769051-7083-0/krb5kdc/kudu_127.6.234.252.keytab
--principal=kudu/127.6.234.252
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.6.234.252:44667
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task1JRg7N/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781955746769051-7083-0/minicluster-data/master-2/data/info.pb
--webserver_interface=127.6.234.252
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.29784
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task1JRg7N/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781955746769051-7083-0/minicluster-data/master-2/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 20 Jun 2026 10:32:37 UTC on ccycloud.quasar-bgyofv.root.comops.site
build id 4457173
I20260620 11:46:49.161219 29784 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 11:46:49.161473 29784 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 11:46:49.166198 29789 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 11:46:49.166234 29790 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 11:46:49.166208 29792 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 11:46:49.166383 29784 server_base.cc:1034] running on GCE node
I20260620 11:46:49.166627 29784 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 11:46:49.166910 29784 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 11:46:49.168076 29784 hybrid_clock.cc:648] HybridClock initialized: now 1781956009168057 us; error 28 us; skew 500 ppm
Jun 20 11:46:49 dist-test-slave-cjc6 krb5kdc[29621](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781956009, etypes {rep=17 tkt=17 ses=17}, kudu/127.6.234.252@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 11:46:49.170825 29784 init.cc:377] Logged in from keytab as kudu/127.6.234.252@KRBTEST.COM (short username kudu)
I20260620 11:46:49.171325 29784 webserver.cc:460] Webserver started at http://127.6.234.252:40471/ using document root <none> and password file <none>
I20260620 11:46:49.171554 29784 fs_manager.cc:359] Metadata directory not provided
I20260620 11:46:49.171628 29784 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 11:46:49.171761 29784 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 11:46:49.172784 29784 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task1JRg7N/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781955746769051-7083-0/minicluster-data/master-2/data/instance:
uuid: "258783154f684f8f8990be1bfb35a7d0"
format_stamp: "Formatted at 2026-06-20 11:46:49 on dist-test-slave-cjc6"
server_key: "55a1e01e6bd6bc6870fb3ba9c0b07da0"
server_key_iv: "77e4288994d23f1f291fd8752029481d"
server_key_version: "encryptionkey@0"
I20260620 11:46:49.173141 29784 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task1JRg7N/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781955746769051-7083-0/minicluster-data/master-2/wal/instance:
uuid: "258783154f684f8f8990be1bfb35a7d0"
format_stamp: "Formatted at 2026-06-20 11:46:49 on dist-test-slave-cjc6"
server_key: "55a1e01e6bd6bc6870fb3ba9c0b07da0"
server_key_iv: "77e4288994d23f1f291fd8752029481d"
server_key_version: "encryptionkey@0"
I20260620 11:46:49.174499 29784 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.003s
I20260620 11:46:49.175177 29799 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 11:46:49.175334 29784 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.001s
I20260620 11:46:49.175428 29784 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task1JRg7N/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781955746769051-7083-0/minicluster-data/master-2/data,/tmp/dist-test-task1JRg7N/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781955746769051-7083-0/minicluster-data/master-2/wal
uuid: "258783154f684f8f8990be1bfb35a7d0"
format_stamp: "Formatted at 2026-06-20 11:46:49 on dist-test-slave-cjc6"
server_key: "55a1e01e6bd6bc6870fb3ba9c0b07da0"
server_key_iv: "77e4288994d23f1f291fd8752029481d"
server_key_version: "encryptionkey@0"
I20260620 11:46:49.175510 29784 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task1JRg7N/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781955746769051-7083-0/minicluster-data/master-2/wal
metadata directory: /tmp/dist-test-task1JRg7N/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781955746769051-7083-0/minicluster-data/master-2/wal
1 data directories: /tmp/dist-test-task1JRg7N/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781955746769051-7083-0/minicluster-data/master-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 11:46:49.182377 29777 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.6.234.252" port: 44667 } attempt: 2
I20260620 11:46:49.182750 29779 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 11:46:49.182615 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.234.252:44667 (local address 127.0.0.1:59488)
0620 11:46:49.182678 (+ 63us) negotiation.cc:107] Waiting for socket to connect
0620 11:46:49.182717 (+ 39us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.234.252:44667: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":37}
I20260620 11:46:49.183086 29779 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 11:46:49.182998 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.234.252:44667 (local address 127.0.0.1:59492)
0620 11:46:49.183042 (+ 44us) negotiation.cc:107] Waiting for socket to connect
0620 11:46:49.183067 (+ 25us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.234.252:44667: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":31}
W20260620 11:46:49.183212 29777 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.6.234.252:44667: Network error: Client connection negotiation failed: client connection to 127.6.234.252:44667: connect: Connection refused (error 111)
I20260620 11:46:49.193310 29784 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 11:46:49.195623 29784 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 11:46:49.195781 29784 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 11:46:49.199847 29784 rpc_server.cc:307] RPC server started. Bound to: 127.6.234.252:44667
I20260620 11:46:49.199934 29851 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.6.234.252:44667 every 8 connection(s)
I20260620 11:46:49.200264 29784 server_base.cc:1166] Dumped server information to /tmp/dist-test-task1JRg7N/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781955746769051-7083-0/minicluster-data/master-2/data/info.pb
I20260620 11:46:49.201076 7083 external_mini_cluster.cc:1388] Started /tmp/dist-test-task1JRg7N/build/release/bin/kudu as pid 29784
I20260620 11:46:49.201215 7083 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task1JRg7N/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781955746769051-7083-0/minicluster-data/master-2/wal/instance
I20260620 11:46:49.201355 29852 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 11:46:49.201463 7083 external_mini_cluster.cc:1428] Setting key 7f8bca3441fc96425ad11183ea9a578a
I20260620 11:46:49.203454 29852 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.6.234.254" port: 38981 } has no permanent_uuid. Determining permanent_uuid...
Jun 20 11:46:49 dist-test-slave-cjc6 krb5kdc[29621](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781956009, etypes {rep=17 tkt=17 ses=17}, kudu/127.6.234.252@KRBTEST.COM for kudu/127.6.234.254@KRBTEST.COM
Jun 20 11:46:49 dist-test-slave-cjc6 krb5kdc[29621](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781956008, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.6.234.252@KRBTEST.COM
I20260620 11:46:49.211915 29706 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 11:46:49.204120 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51262 (local address 127.6.234.254:38981)
0620 11:46:49.204184 (+ 64us) server_negotiation.cc:206] Beginning negotiation
0620 11:46:49.204187 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0620 11:46:49.204319 (+ 132us) server_negotiation.cc:407] Connection header received
0620 11:46:49.204363 (+ 44us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 11:46:49.204364 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 11:46:49.204374 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 11:46:49.204434 (+ 60us) server_negotiation.cc:226] Negotiated authn=SASL
0620 11:46:49.204868 (+ 434us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 11:46:49.205584 (+ 716us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 11:46:49.206396 (+ 812us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 11:46:49.206593 (+ 197us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 11:46:49.206678 (+ 85us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 11:46:49.208619 (+ 1941us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 11:46:49.208620 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 11:46:49.208621 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 11:46:49.208627 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0620 11:46:49.210644 (+ 2017us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 11:46:49.211045 (+ 401us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 11:46:49.211046 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 11:46:49.211046 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 11:46:49.211105 (+ 59us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 11:46:49.211327 (+ 222us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 11:46:49.211328 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 11:46:49.211329 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 11:46:49.211452 (+ 123us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 11:46:49.211530 (+ 78us) server_negotiation.cc:1035] Waiting for connection context
0620 11:46:49.211745 (+ 215us) server_negotiation.cc:299] Negotiation successful
0620 11:46:49.211823 (+ 78us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":35}
I20260620 11:46:49.212126 29856 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 11:46:49.204033 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.234.254:38981 (local address 127.0.0.1:51262)
0620 11:46:49.204210 (+ 177us) negotiation.cc:107] Waiting for socket to connect
0620 11:46:49.204247 (+ 37us) client_negotiation.cc:174] Beginning negotiation
0620 11:46:49.204315 (+ 68us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 11:46:49.204457 (+ 142us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 11:46:49.204458 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 11:46:49.204692 (+ 234us) client_negotiation.cc:189] Negotiated authn=SASL
0620 11:46:49.204787 (+ 95us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 11:46:49.204791 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 11:46:49.205665 (+ 874us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 11:46:49.205665 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 11:46:49.206305 (+ 640us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 11:46:49.206307 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 11:46:49.206712 (+ 405us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 11:46:49.206713 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 11:46:49.206791 (+ 78us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 11:46:49.207132 (+ 341us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 11:46:49.207139 (+ 7us) client_negotiation.cc:656] calling sasl_client_start()
0620 11:46:49.208524 (+ 1385us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 11:46:49.210756 (+ 2232us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 11:46:49.210762 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 11:46:49.210764 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0620 11:46:49.210955 (+ 191us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 11:46:49.211171 (+ 216us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 11:46:49.211172 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 11:46:49.211172 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0620 11:46:49.211261 (+ 89us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 11:46:49.211521 (+ 260us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 11:46:49.211527 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 11:46:49.211625 (+ 98us) client_negotiation.cc:769] Sending connection context
0620 11:46:49.211723 (+ 98us) client_negotiation.cc:240] Negotiation successful
0620 11:46:49.211874 (+ 151us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":151,"thread_start_us":103,"threads_started":1}
I20260620 11:46:49.212131 29855 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 11:46:49.201975 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59508 (local address 127.6.234.252:44667)
0620 11:46:49.202163 (+ 188us) server_negotiation.cc:206] Beginning negotiation
0620 11:46:49.202173 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0620 11:46:49.202204 (+ 31us) server_negotiation.cc:407] Connection header received
0620 11:46:49.202826 (+ 622us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 11:46:49.202832 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 11:46:49.202881 (+ 49us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 11:46:49.203034 (+ 153us) server_negotiation.cc:226] Negotiated authn=SASL
0620 11:46:49.203665 (+ 631us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 11:46:49.204806 (+ 1141us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 11:46:49.205878 (+ 1072us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 11:46:49.206285 (+ 407us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 11:46:49.206413 (+ 128us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 11:46:49.209079 (+ 2666us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 11:46:49.209086 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 11:46:49.209087 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 11:46:49.209094 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0620 11:46:49.210726 (+ 1632us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 11:46:49.211109 (+ 383us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 11:46:49.211113 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 11:46:49.211113 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 11:46:49.211201 (+ 88us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 11:46:49.211420 (+ 219us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 11:46:49.211421 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 11:46:49.211422 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 11:46:49.211625 (+ 203us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 11:46:49.211773 (+ 148us) server_negotiation.cc:1035] Waiting for connection context
0620 11:46:49.211861 (+ 88us) server_negotiation.cc:299] Negotiation successful
0620 11:46:49.211939 (+ 78us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":137,"thread_start_us":83,"threads_started":1}
I20260620 11:46:49.212613 29852 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.6.234.253" port: 36053 } has no permanent_uuid. Determining permanent_uuid...
Jun 20 11:46:49 dist-test-slave-cjc6 krb5kdc[29621](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781956009, etypes {rep=17 tkt=17 ses=17}, kudu/127.6.234.252@KRBTEST.COM for kudu/127.6.234.253@KRBTEST.COM
Jun 20 11:46:49 dist-test-slave-cjc6 krb5kdc[29621](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781956008, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.6.234.254@KRBTEST.COM
I20260620 11:46:49.219025 29781 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 11:46:49.213010 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48920 (local address 127.6.234.253:36053)
0620 11:46:49.213056 (+ 46us) server_negotiation.cc:206] Beginning negotiation
0620 11:46:49.213058 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 11:46:49.213065 (+ 7us) server_negotiation.cc:407] Connection header received
0620 11:46:49.213088 (+ 23us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 11:46:49.213089 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 11:46:49.213098 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 11:46:49.213170 (+ 72us) server_negotiation.cc:226] Negotiated authn=SASL
0620 11:46:49.213449 (+ 279us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 11:46:49.213987 (+ 538us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 11:46:49.214629 (+ 642us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 11:46:49.214832 (+ 203us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 11:46:49.214905 (+ 73us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 11:46:49.216517 (+ 1612us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 11:46:49.216519 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 11:46:49.216519 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 11:46:49.216526 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0620 11:46:49.217909 (+ 1383us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 11:46:49.218251 (+ 342us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 11:46:49.218252 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 11:46:49.218253 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 11:46:49.218322 (+ 69us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 11:46:49.218526 (+ 204us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 11:46:49.218526 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 11:46:49.218527 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 11:46:49.218636 (+ 109us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 11:46:49.218705 (+ 69us) server_negotiation.cc:1035] Waiting for connection context
0620 11:46:49.218868 (+ 163us) server_negotiation.cc:299] Negotiation successful
0620 11:46:49.218940 (+ 72us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":25}
I20260620 11:46:49.219024 29856 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 11:46:49.212924 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.234.253:36053 (local address 127.0.0.1:48920)
0620 11:46:49.212956 (+ 32us) negotiation.cc:107] Waiting for socket to connect
0620 11:46:49.212965 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0620 11:46:49.213002 (+ 37us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 11:46:49.213172 (+ 170us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 11:46:49.213173 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 11:46:49.213327 (+ 154us) client_negotiation.cc:189] Negotiated authn=SASL
0620 11:46:49.213370 (+ 43us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 11:46:49.213371 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 11:46:49.214062 (+ 691us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 11:46:49.214063 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 11:46:49.214538 (+ 475us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 11:46:49.214540 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 11:46:49.214901 (+ 361us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 11:46:49.214902 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 11:46:49.214967 (+ 65us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 11:46:49.215327 (+ 360us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 11:46:49.215331 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0620 11:46:49.216430 (+ 1099us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 11:46:49.218001 (+ 1571us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 11:46:49.218002 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 11:46:49.218002 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0620 11:46:49.218166 (+ 164us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 11:46:49.218388 (+ 222us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 11:46:49.218389 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 11:46:49.218389 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0620 11:46:49.218445 (+ 56us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 11:46:49.218711 (+ 266us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 11:46:49.218712 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 11:46:49.218751 (+ 39us) client_negotiation.cc:769] Sending connection context
0620 11:46:49.218843 (+ 92us) client_negotiation.cc:240] Negotiation successful
0620 11:46:49.218933 (+ 90us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":17}
I20260620 11:46:49.219498 29852 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.6.234.252" port: 44667 } has no permanent_uuid. Determining permanent_uuid...
I20260620 11:46:49.220427 29706 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 11:46:49.213225 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51264 (local address 127.6.234.254:38981)
0620 11:46:49.213284 (+ 59us) server_negotiation.cc:206] Beginning negotiation
0620 11:46:49.213286 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 11:46:49.213292 (+ 6us) server_negotiation.cc:407] Connection header received
0620 11:46:49.213306 (+ 14us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 11:46:49.213307 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 11:46:49.213314 (+ 7us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 11:46:49.213371 (+ 57us) server_negotiation.cc:226] Negotiated authn=SASL
0620 11:46:49.213903 (+ 532us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 11:46:49.214339 (+ 436us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 11:46:49.215009 (+ 670us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 11:46:49.215301 (+ 292us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 11:46:49.215374 (+ 73us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 11:46:49.217816 (+ 2442us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 11:46:49.217818 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 11:46:49.217818 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 11:46:49.217825 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0620 11:46:49.219253 (+ 1428us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 11:46:49.219621 (+ 368us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 11:46:49.219622 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 11:46:49.219622 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 11:46:49.219679 (+ 57us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 11:46:49.219921 (+ 242us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 11:46:49.219922 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 11:46:49.219922 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 11:46:49.220057 (+ 135us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 11:46:49.220216 (+ 159us) server_negotiation.cc:1035] Waiting for connection context
0620 11:46:49.220286 (+ 70us) server_negotiation.cc:299] Negotiation successful
0620 11:46:49.220346 (+ 60us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":43}
I20260620 11:46:49.221990 29702 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.6.234.252" port: 44667 } attempt: 2
Jun 20 11:46:49 dist-test-slave-cjc6 krb5kdc[29621](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781956009, etypes {rep=17 tkt=17 ses=17}, kudu/127.6.234.252@KRBTEST.COM for kudu/127.6.234.252@KRBTEST.COM
Jun 20 11:46:49 dist-test-slave-cjc6 krb5kdc[29621](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781956008, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.6.234.253@KRBTEST.COM
I20260620 11:46:49.226300 29856 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 11:46:49.219757 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.234.252:44667 (local address 127.0.0.1:59524)
0620 11:46:49.219799 (+ 42us) negotiation.cc:107] Waiting for socket to connect
0620 11:46:49.219809 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0620 11:46:49.219842 (+ 33us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 11:46:49.219950 (+ 108us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 11:46:49.219951 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 11:46:49.220091 (+ 140us) client_negotiation.cc:189] Negotiated authn=SASL
0620 11:46:49.220124 (+ 33us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 11:46:49.220125 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 11:46:49.220958 (+ 833us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 11:46:49.220959 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 11:46:49.221498 (+ 539us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 11:46:49.221500 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 11:46:49.221945 (+ 445us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 11:46:49.221946 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 11:46:49.221991 (+ 45us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 11:46:49.222405 (+ 414us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 11:46:49.222410 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0620 11:46:49.223678 (+ 1268us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 11:46:49.225278 (+ 1600us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 11:46:49.225279 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 11:46:49.225279 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0620 11:46:49.225437 (+ 158us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 11:46:49.225653 (+ 216us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 11:46:49.225654 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 11:46:49.225654 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0620 11:46:49.225741 (+ 87us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 11:46:49.225975 (+ 234us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 11:46:49.225976 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 11:46:49.226011 (+ 35us) client_negotiation.cc:769] Sending connection context
0620 11:46:49.226101 (+ 90us) client_negotiation.cc:240] Negotiation successful
0620 11:46:49.226191 (+ 90us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":25}
I20260620 11:46:49.226491 29855 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 11:46:49.219799 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59524 (local address 127.6.234.252:44667)
0620 11:46:49.219835 (+ 36us) server_negotiation.cc:206] Beginning negotiation
0620 11:46:49.219836 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0620 11:46:49.219843 (+ 7us) server_negotiation.cc:407] Connection header received
0620 11:46:49.219886 (+ 43us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 11:46:49.219887 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 11:46:49.219897 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 11:46:49.219946 (+ 49us) server_negotiation.cc:226] Negotiated authn=SASL
0620 11:46:49.220193 (+ 247us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 11:46:49.220883 (+ 690us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 11:46:49.221592 (+ 709us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 11:46:49.221865 (+ 273us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 11:46:49.221946 (+ 81us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 11:46:49.223752 (+ 1806us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 11:46:49.223753 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 11:46:49.223753 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 11:46:49.223766 (+ 13us) server_negotiation.cc:922] calling sasl_server_start()
0620 11:46:49.225198 (+ 1432us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 11:46:49.225520 (+ 322us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 11:46:49.225521 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 11:46:49.225521 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 11:46:49.225567 (+ 46us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 11:46:49.225791 (+ 224us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 11:46:49.225792 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 11:46:49.225793 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 11:46:49.225896 (+ 103us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 11:46:49.225978 (+ 82us) server_negotiation.cc:1035] Waiting for connection context
0620 11:46:49.226123 (+ 145us) server_negotiation.cc:299] Negotiation successful
0620 11:46:49.226446 (+ 323us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":253,"server-negotiator.queue_time_us":20}
Jun 20 11:46:49 dist-test-slave-cjc6 krb5kdc[29621](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781956008, etypes {rep=17 tkt=17 ses=17}, kudu/127.6.234.254@KRBTEST.COM for kudu/127.6.234.252@KRBTEST.COM
I20260620 11:46:49.228570 29852 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 258783154f684f8f8990be1bfb35a7d0: Bootstrap starting.
I20260620 11:46:49.229384 29852 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 258783154f684f8f8990be1bfb35a7d0: Neither blocks nor log segments found. Creating new log.
I20260620 11:46:49.229668 29781 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 11:46:49.221011 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48928 (local address 127.6.234.253:36053)
0620 11:46:49.221048 (+ 37us) server_negotiation.cc:206] Beginning negotiation
0620 11:46:49.221050 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 11:46:49.221058 (+ 8us) server_negotiation.cc:407] Connection header received
0620 11:46:49.221097 (+ 39us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 11:46:49.221098 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 11:46:49.221107 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 11:46:49.221161 (+ 54us) server_negotiation.cc:226] Negotiated authn=SASL
0620 11:46:49.221603 (+ 442us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 11:46:49.222188 (+ 585us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 11:46:49.223038 (+ 850us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 11:46:49.223410 (+ 372us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 11:46:49.223471 (+ 61us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 11:46:49.226183 (+ 2712us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 11:46:49.226184 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 11:46:49.226185 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 11:46:49.226191 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0620 11:46:49.227927 (+ 1736us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 11:46:49.228634 (+ 707us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 11:46:49.228635 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 11:46:49.228635 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 11:46:49.228693 (+ 58us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 11:46:49.228898 (+ 205us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 11:46:49.228899 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 11:46:49.228899 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 11:46:49.229040 (+ 141us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 11:46:49.229188 (+ 148us) server_negotiation.cc:1035] Waiting for connection context
0620 11:46:49.229422 (+ 234us) server_negotiation.cc:299] Negotiation successful
0620 11:46:49.229557 (+ 135us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":19}
I20260620 11:46:49.229750 29852 log.cc:824] T 00000000000000000000000000000000 P 258783154f684f8f8990be1bfb35a7d0: Log is configured to *not* fsync() on all Append() calls
I20260620 11:46:49.230216 29705 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 11:46:49.222229 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.234.252:44667 (local address 127.0.0.1:59528)
0620 11:46:49.222289 (+ 60us) negotiation.cc:107] Waiting for socket to connect
0620 11:46:49.222298 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0620 11:46:49.222333 (+ 35us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 11:46:49.222713 (+ 380us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 11:46:49.222714 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 11:46:49.222868 (+ 154us) client_negotiation.cc:189] Negotiated authn=SASL
0620 11:46:49.222905 (+ 37us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 11:46:49.222906 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 11:46:49.224167 (+ 1261us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 11:46:49.224168 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 11:46:49.224821 (+ 653us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 11:46:49.224824 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 11:46:49.225337 (+ 513us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 11:46:49.225338 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 11:46:49.225404 (+ 66us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 11:46:49.225824 (+ 420us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 11:46:49.225829 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0620 11:46:49.227161 (+ 1332us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 11:46:49.229095 (+ 1934us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 11:46:49.229096 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 11:46:49.229097 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0620 11:46:49.229272 (+ 175us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 11:46:49.229493 (+ 221us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 11:46:49.229494 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 11:46:49.229494 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0620 11:46:49.229553 (+ 59us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 11:46:49.229905 (+ 352us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 11:46:49.229906 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 11:46:49.229955 (+ 49us) client_negotiation.cc:769] Sending connection context
0620 11:46:49.230031 (+ 76us) client_negotiation.cc:240] Negotiation successful
0620 11:46:49.230124 (+ 93us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":46}
I20260620 11:46:49.230646 29857 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 11:46:49.222408 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59528 (local address 127.6.234.252:44667)
0620 11:46:49.222585 (+ 177us) server_negotiation.cc:206] Beginning negotiation
0620 11:46:49.222587 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 11:46:49.222594 (+ 7us) server_negotiation.cc:407] Connection header received
0620 11:46:49.222611 (+ 17us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 11:46:49.222612 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 11:46:49.222622 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 11:46:49.222690 (+ 68us) server_negotiation.cc:226] Negotiated authn=SASL
0620 11:46:49.222996 (+ 306us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 11:46:49.224067 (+ 1071us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 11:46:49.224919 (+ 852us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 11:46:49.225237 (+ 318us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 11:46:49.226346 (+ 1109us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 11:46:49.227274 (+ 928us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 11:46:49.227275 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 11:46:49.227276 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 11:46:49.227282 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0620 11:46:49.229021 (+ 1739us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 11:46:49.229358 (+ 337us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 11:46:49.229358 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 11:46:49.229359 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 11:46:49.229415 (+ 56us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 11:46:49.229625 (+ 210us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 11:46:49.229625 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 11:46:49.229626 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 11:46:49.229813 (+ 187us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 11:46:49.229899 (+ 86us) server_negotiation.cc:1035] Waiting for connection context
0620 11:46:49.230407 (+ 508us) server_negotiation.cc:299] Negotiation successful
0620 11:46:49.230504 (+ 97us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":156,"thread_start_us":106,"threads_started":1}
I20260620 11:46:49.230806 29852 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 258783154f684f8f8990be1bfb35a7d0: No bootstrap required, opened a new log
I20260620 11:46:49.231395 7083 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260620 11:46:49.232182 29702 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P ecbaec1ef2a8438da898ed54ae0773a2: Bootstrap starting.
I20260620 11:46:49.232623 29852 raft_consensus.cc:348] T 00000000000000000000000000000000 P 258783154f684f8f8990be1bfb35a7d0 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ecbaec1ef2a8438da898ed54ae0773a2" member_type: VOTER last_known_addr { host: "127.6.234.254" port: 38981 } } peers { permanent_uuid: "02ab4448e0714124b680d996d3cd0f8a" member_type: VOTER last_known_addr { host: "127.6.234.253" port: 36053 } } peers { permanent_uuid: "258783154f684f8f8990be1bfb35a7d0" member_type: VOTER last_known_addr { host: "127.6.234.252" port: 44667 } }
I20260620 11:46:49.232738 29702 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P ecbaec1ef2a8438da898ed54ae0773a2: Neither blocks nor log segments found. Creating new log.
I20260620 11:46:49.232769 29852 raft_consensus.cc:729] T 00000000000000000000000000000000 P 258783154f684f8f8990be1bfb35a7d0 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 258783154f684f8f8990be1bfb35a7d0, State: Initialized, Role: FOLLOWER
I20260620 11:46:49.232846 29852 consensus_queue.cc:260] T 00000000000000000000000000000000 P 258783154f684f8f8990be1bfb35a7d0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ecbaec1ef2a8438da898ed54ae0773a2" member_type: VOTER last_known_addr { host: "127.6.234.254" port: 38981 } } peers { permanent_uuid: "02ab4448e0714124b680d996d3cd0f8a" member_type: VOTER last_known_addr { host: "127.6.234.253" port: 36053 } } peers { permanent_uuid: "258783154f684f8f8990be1bfb35a7d0" member_type: VOTER last_known_addr { host: "127.6.234.252" port: 44667 } }
I20260620 11:46:49.232954 29702 log.cc:824] T 00000000000000000000000000000000 P ecbaec1ef2a8438da898ed54ae0773a2: Log is configured to *not* fsync() on all Append() calls
I20260620 11:46:49.233150 29852 sys_catalog.cc:564] T 00000000000000000000000000000000 P 258783154f684f8f8990be1bfb35a7d0 [sys.catalog]: configured and running, proceeding with master startup.
I20260620 11:46:49.233111 29859 sys_catalog.cc:455] T 00000000000000000000000000000000 P 258783154f684f8f8990be1bfb35a7d0 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ecbaec1ef2a8438da898ed54ae0773a2" member_type: VOTER last_known_addr { host: "127.6.234.254" port: 38981 } } peers { permanent_uuid: "02ab4448e0714124b680d996d3cd0f8a" member_type: VOTER last_known_addr { host: "127.6.234.253" port: 36053 } } peers { permanent_uuid: "258783154f684f8f8990be1bfb35a7d0" member_type: VOTER last_known_addr { host: "127.6.234.252" port: 44667 } } }
I20260620 11:46:49.233209 29859 sys_catalog.cc:458] T 00000000000000000000000000000000 P 258783154f684f8f8990be1bfb35a7d0 [sys.catalog]: This master's current role is: FOLLOWER
I20260620 11:46:49.232350 29676 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "00000000000000000000000000000000"
dest_uuid: "ecbaec1ef2a8438da898ed54ae0773a2"
from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:51264
I20260620 11:46:49.233762 29702 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P ecbaec1ef2a8438da898ed54ae0773a2: No bootstrap required, opened a new log
I20260620 11:46:49.234974 29702 raft_consensus.cc:348] T 00000000000000000000000000000000 P ecbaec1ef2a8438da898ed54ae0773a2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ecbaec1ef2a8438da898ed54ae0773a2" member_type: VOTER last_known_addr { host: "127.6.234.254" port: 38981 } } peers { permanent_uuid: "02ab4448e0714124b680d996d3cd0f8a" member_type: VOTER last_known_addr { host: "127.6.234.253" port: 36053 } } peers { permanent_uuid: "258783154f684f8f8990be1bfb35a7d0" member_type: VOTER last_known_addr { host: "127.6.234.252" port: 44667 } }
I20260620 11:46:49.235107 29702 raft_consensus.cc:729] T 00000000000000000000000000000000 P ecbaec1ef2a8438da898ed54ae0773a2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ecbaec1ef2a8438da898ed54ae0773a2, State: Initialized, Role: FOLLOWER
I20260620 11:46:49.235181 29702 consensus_queue.cc:260] T 00000000000000000000000000000000 P ecbaec1ef2a8438da898ed54ae0773a2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ecbaec1ef2a8438da898ed54ae0773a2" member_type: VOTER last_known_addr { host: "127.6.234.254" port: 38981 } } peers { permanent_uuid: "02ab4448e0714124b680d996d3cd0f8a" member_type: VOTER last_known_addr { host: "127.6.234.253" port: 36053 } } peers { permanent_uuid: "258783154f684f8f8990be1bfb35a7d0" member_type: VOTER last_known_addr { host: "127.6.234.252" port: 44667 } }
I20260620 11:46:49.235458 29702 sys_catalog.cc:564] T 00000000000000000000000000000000 P ecbaec1ef2a8438da898ed54ae0773a2 [sys.catalog]: configured and running, proceeding with master startup.
W20260620 11:46:49.235471 29871 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 258783154f684f8f8990be1bfb35a7d0: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260620 11:46:49.235584 29871 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
W20260620 11:46:49.237195 29884 catalog_manager.cc:1558] T 00000000000000000000000000000000 P ecbaec1ef2a8438da898ed54ae0773a2: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260620 11:46:49.237313 29884 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260620 11:46:49.235317 29784 master_runner.cc:186] Error getting master registration for 127.6.234.254:38981: OK, instance_id { permanent_uuid: "ecbaec1ef2a8438da898ed54ae0773a2" instance_seqno: 1781956008869203 } error { code: CATALOG_MANAGER_NOT_INITIALIZED status { code: SERVICE_UNAVAILABLE message: "Catalog manager is not initialized. State: Starting" } }
I20260620 11:46:49.237649 29784 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20260620 11:46:49.237746 29676 raft_consensus.cc:482] T 00000000000000000000000000000000 P ecbaec1ef2a8438da898ed54ae0773a2 [term 0 FOLLOWER]: Starting forced leader election (received explicit request)
I20260620 11:46:49.237797 29676 raft_consensus.cc:3037] T 00000000000000000000000000000000 P ecbaec1ef2a8438da898ed54ae0773a2 [term 0 FOLLOWER]: Advancing to term 1
I20260620 11:46:49.238442 29676 raft_consensus.cc:504] T 00000000000000000000000000000000 P ecbaec1ef2a8438da898ed54ae0773a2 [term 1 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ecbaec1ef2a8438da898ed54ae0773a2" member_type: VOTER last_known_addr { host: "127.6.234.254" port: 38981 } } peers { permanent_uuid: "02ab4448e0714124b680d996d3cd0f8a" member_type: VOTER last_known_addr { host: "127.6.234.253" port: 36053 } } peers { permanent_uuid: "258783154f684f8f8990be1bfb35a7d0" member_type: VOTER last_known_addr { host: "127.6.234.252" port: 44667 } }
I20260620 11:46:49.238699 29676 leader_election.cc:288] T 00000000000000000000000000000000 P ecbaec1ef2a8438da898ed54ae0773a2 [CANDIDATE]: Term 1 election: Requested vote from peers 02ab4448e0714124b680d996d3cd0f8a (127.6.234.253:36053), 258783154f684f8f8990be1bfb35a7d0 (127.6.234.252:44667)
I20260620 11:46:49.237978 29634 master_runner.cc:186] Error getting master registration for 127.6.234.253:36053: OK, instance_id { permanent_uuid: "02ab4448e0714124b680d996d3cd0f8a" instance_seqno: 1781956009034596 } error { code: CATALOG_MANAGER_NOT_INITIALIZED status { code: SERVICE_UNAVAILABLE message: "Catalog manager is not initialized. State: Starting" } }
I20260620 11:46:49.240803 29634 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20260620 11:46:49.241068 29826 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "ecbaec1ef2a8438da898ed54ae0773a2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: true dest_uuid: "258783154f684f8f8990be1bfb35a7d0"
I20260620 11:46:49.241107 29873 sys_catalog.cc:455] T 00000000000000000000000000000000 P ecbaec1ef2a8438da898ed54ae0773a2 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ecbaec1ef2a8438da898ed54ae0773a2" member_type: VOTER last_known_addr { host: "127.6.234.254" port: 38981 } } peers { permanent_uuid: "02ab4448e0714124b680d996d3cd0f8a" member_type: VOTER last_known_addr { host: "127.6.234.253" port: 36053 } } peers { permanent_uuid: "258783154f684f8f8990be1bfb35a7d0" member_type: VOTER last_known_addr { host: "127.6.234.252" port: 44667 } } }
I20260620 11:46:49.241178 29873 sys_catalog.cc:458] T 00000000000000000000000000000000 P ecbaec1ef2a8438da898ed54ae0773a2 [sys.catalog]: This master's current role is: FOLLOWER
I20260620 11:46:49.241185 29826 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 258783154f684f8f8990be1bfb35a7d0 [term 0 FOLLOWER]: Advancing to term 1
I20260620 11:46:49.242043 29826 raft_consensus.cc:2454] T 00000000000000000000000000000000 P 258783154f684f8f8990be1bfb35a7d0 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ecbaec1ef2a8438da898ed54ae0773a2 in term 1.
I20260620 11:46:49.242257 29650 leader_election.cc:302] T 00000000000000000000000000000000 P ecbaec1ef2a8438da898ed54ae0773a2 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 258783154f684f8f8990be1bfb35a7d0, ecbaec1ef2a8438da898ed54ae0773a2; no voters:
I20260620 11:46:49.242367 29873 raft_consensus.cc:2781] T 00000000000000000000000000000000 P ecbaec1ef2a8438da898ed54ae0773a2 [term 1 FOLLOWER]: Leader election won for term 1
I20260620 11:46:49.242542 29873 raft_consensus.cc:686] T 00000000000000000000000000000000 P ecbaec1ef2a8438da898ed54ae0773a2 [term 1 LEADER]: Becoming Leader. State: Replica: ecbaec1ef2a8438da898ed54ae0773a2, State: Running, Role: LEADER
I20260620 11:46:49.242630 29873 consensus_queue.cc:237] T 00000000000000000000000000000000 P ecbaec1ef2a8438da898ed54ae0773a2 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ecbaec1ef2a8438da898ed54ae0773a2" member_type: VOTER last_known_addr { host: "127.6.234.254" port: 38981 } } peers { permanent_uuid: "02ab4448e0714124b680d996d3cd0f8a" member_type: VOTER last_known_addr { host: "127.6.234.253" port: 36053 } } peers { permanent_uuid: "258783154f684f8f8990be1bfb35a7d0" member_type: VOTER last_known_addr { host: "127.6.234.252" port: 44667 } }
I20260620 11:46:49.249805 29893 sys_catalog.cc:455] T 00000000000000000000000000000000 P ecbaec1ef2a8438da898ed54ae0773a2 [sys.catalog]: SysCatalogTable state changed. Reason: New leader ecbaec1ef2a8438da898ed54ae0773a2. Latest consensus state: current_term: 1 leader_uuid: "ecbaec1ef2a8438da898ed54ae0773a2" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ecbaec1ef2a8438da898ed54ae0773a2" member_type: VOTER last_known_addr { host: "127.6.234.254" port: 38981 } } peers { permanent_uuid: "02ab4448e0714124b680d996d3cd0f8a" member_type: VOTER last_known_addr { host: "127.6.234.253" port: 36053 } } peers { permanent_uuid: "258783154f684f8f8990be1bfb35a7d0" member_type: VOTER last_known_addr { host: "127.6.234.252" port: 44667 } } }
I20260620 11:46:49.249915 29893 sys_catalog.cc:458] T 00000000000000000000000000000000 P ecbaec1ef2a8438da898ed54ae0773a2 [sys.catalog]: This master's current role is: LEADER
I20260620 11:46:49.250094 29895 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260620 11:46:49.250260 29895 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260620 11:46:49.258709 29706 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 11:46:49.240207 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51280 (local address 127.6.234.254:38981)
0620 11:46:49.240252 (+ 45us) server_negotiation.cc:206] Beginning negotiation
0620 11:46:49.240254 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 11:46:49.249925 (+ 9671us) server_negotiation.cc:407] Connection header received
0620 11:46:49.250108 (+ 183us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 11:46:49.250109 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 11:46:49.250120 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 11:46:49.250153 (+ 33us) server_negotiation.cc:226] Negotiated authn=SASL
0620 11:46:49.251006 (+ 853us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 11:46:49.251756 (+ 750us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 11:46:49.252798 (+ 1042us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 11:46:49.253097 (+ 299us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 11:46:49.253136 (+ 39us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 11:46:49.255235 (+ 2099us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 11:46:49.255237 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 11:46:49.255237 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 11:46:49.255244 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0620 11:46:49.257055 (+ 1811us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 11:46:49.257620 (+ 565us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 11:46:49.257621 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 11:46:49.257621 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 11:46:49.257677 (+ 56us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 11:46:49.258017 (+ 340us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 11:46:49.258018 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 11:46:49.258018 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 11:46:49.258153 (+ 135us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 11:46:49.258241 (+ 88us) server_negotiation.cc:1035] Waiting for connection context
0620 11:46:49.258545 (+ 304us) server_negotiation.cc:299] Negotiation successful
0620 11:46:49.258619 (+ 74us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":23}
I20260620 11:46:49.262656 29826 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 258783154f684f8f8990be1bfb35a7d0 [term 1 FOLLOWER]: Refusing update from remote peer ecbaec1ef2a8438da898ed54ae0773a2: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 11:46:49.263082 29893 consensus_queue.cc:1035] T 00000000000000000000000000000000 P ecbaec1ef2a8438da898ed54ae0773a2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "258783154f684f8f8990be1bfb35a7d0" member_type: VOTER last_known_addr { host: "127.6.234.252" port: 44667 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 11:46:49.263105 29781 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 11:46:49.239924 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48944 (local address 127.6.234.253:36053)
0620 11:46:49.240211 (+ 287us) server_negotiation.cc:206] Beginning negotiation
0620 11:46:49.240214 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0620 11:46:49.249997 (+ 9783us) server_negotiation.cc:407] Connection header received
0620 11:46:49.250132 (+ 135us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 11:46:49.250133 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 11:46:49.250144 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 11:46:49.250174 (+ 30us) server_negotiation.cc:226] Negotiated authn=SASL
0620 11:46:49.251225 (+ 1051us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 11:46:49.251905 (+ 680us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 11:46:49.254631 (+ 2726us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 11:46:49.255442 (+ 811us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 11:46:49.255740 (+ 298us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 11:46:49.259063 (+ 3323us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 11:46:49.259065 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 11:46:49.259065 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 11:46:49.259072 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0620 11:46:49.261082 (+ 2010us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 11:46:49.261886 (+ 804us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 11:46:49.261887 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 11:46:49.261887 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 11:46:49.261958 (+ 71us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 11:46:49.262270 (+ 312us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 11:46:49.262272 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 11:46:49.262272 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 11:46:49.262420 (+ 148us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 11:46:49.262509 (+ 89us) server_negotiation.cc:1035] Waiting for connection context
0620 11:46:49.262929 (+ 420us) server_negotiation.cc:299] Negotiation successful
0620 11:46:49.263003 (+ 74us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":263}
I20260620 11:46:49.264410 29859 sys_catalog.cc:455] T 00000000000000000000000000000000 P 258783154f684f8f8990be1bfb35a7d0 [sys.catalog]: SysCatalogTable state changed. Reason: New leader ecbaec1ef2a8438da898ed54ae0773a2. Latest consensus state: current_term: 1 leader_uuid: "ecbaec1ef2a8438da898ed54ae0773a2" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ecbaec1ef2a8438da898ed54ae0773a2" member_type: VOTER last_known_addr { host: "127.6.234.254" port: 38981 } } peers { permanent_uuid: "02ab4448e0714124b680d996d3cd0f8a" member_type: VOTER last_known_addr { host: "127.6.234.253" port: 36053 } } peers { permanent_uuid: "258783154f684f8f8990be1bfb35a7d0" member_type: VOTER last_known_addr { host: "127.6.234.252" port: 44667 } } }
I20260620 11:46:49.264523 29859 sys_catalog.cc:458] T 00000000000000000000000000000000 P 258783154f684f8f8990be1bfb35a7d0 [sys.catalog]: This master's current role is: FOLLOWER
I20260620 11:46:49.265386 29896 mvcc.cc:205] Tried to move back new op lower bound from 7298891813935972352 to 7298891813858541568. Current Snapshot: MvccSnapshot[applied={T|T < 7298891813935972352}]
I20260620 11:46:49.265455 29873 sys_catalog.cc:455] T 00000000000000000000000000000000 P ecbaec1ef2a8438da898ed54ae0773a2 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "ecbaec1ef2a8438da898ed54ae0773a2" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ecbaec1ef2a8438da898ed54ae0773a2" member_type: VOTER last_known_addr { host: "127.6.234.254" port: 38981 } } peers { permanent_uuid: "02ab4448e0714124b680d996d3cd0f8a" member_type: VOTER last_known_addr { host: "127.6.234.253" port: 36053 } } peers { permanent_uuid: "258783154f684f8f8990be1bfb35a7d0" member_type: VOTER last_known_addr { host: "127.6.234.252" port: 44667 } } }
I20260620 11:46:49.265527 29873 sys_catalog.cc:458] T 00000000000000000000000000000000 P ecbaec1ef2a8438da898ed54ae0773a2 [sys.catalog]: This master's current role is: LEADER
I20260620 11:46:49.265511 29859 sys_catalog.cc:455] T 00000000000000000000000000000000 P 258783154f684f8f8990be1bfb35a7d0 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "ecbaec1ef2a8438da898ed54ae0773a2" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ecbaec1ef2a8438da898ed54ae0773a2" member_type: VOTER last_known_addr { host: "127.6.234.254" port: 38981 } } peers { permanent_uuid: "02ab4448e0714124b680d996d3cd0f8a" member_type: VOTER last_known_addr { host: "127.6.234.253" port: 36053 } } peers { permanent_uuid: "258783154f684f8f8990be1bfb35a7d0" member_type: VOTER last_known_addr { host: "127.6.234.252" port: 44667 } } }
I20260620 11:46:49.265583 29859 sys_catalog.cc:458] T 00000000000000000000000000000000 P 258783154f684f8f8990be1bfb35a7d0 [sys.catalog]: This master's current role is: FOLLOWER
I20260620 11:46:49.266173 29897 mvcc.cc:205] Tried to move back new op lower bound from 7298891813935972352 to 7298891813858541568. Current Snapshot: MvccSnapshot[applied={T|T < 7298891813935972352}]
I20260620 11:46:49.266649 29857 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 11:46:49.240268 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59540 (local address 127.6.234.252:44667)
0620 11:46:49.240302 (+ 34us) server_negotiation.cc:206] Beginning negotiation
0620 11:46:49.240304 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 11:46:49.252916 (+ 12612us) server_negotiation.cc:407] Connection header received
0620 11:46:49.253117 (+ 201us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 11:46:49.253118 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 11:46:49.253148 (+ 30us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 11:46:49.253179 (+ 31us) server_negotiation.cc:226] Negotiated authn=SASL
0620 11:46:49.257339 (+ 4160us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 11:46:49.258100 (+ 761us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 11:46:49.260011 (+ 1911us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 11:46:49.260350 (+ 339us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 11:46:49.260389 (+ 39us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 11:46:49.262944 (+ 2555us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 11:46:49.262945 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 11:46:49.262945 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 11:46:49.262952 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0620 11:46:49.264543 (+ 1591us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 11:46:49.264926 (+ 383us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 11:46:49.264927 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 11:46:49.264928 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 11:46:49.264981 (+ 53us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 11:46:49.265632 (+ 651us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 11:46:49.265634 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 11:46:49.265634 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 11:46:49.265779 (+ 145us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 11:46:49.266265 (+ 486us) server_negotiation.cc:1035] Waiting for connection context
0620 11:46:49.266465 (+ 200us) server_negotiation.cc:299] Negotiation successful
0620 11:46:49.266546 (+ 81us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":17}
I20260620 11:46:49.266641 29895 catalog_manager.cc:1347] Generated new cluster ID: 8679e78cd6c64fd2a94ab66fe13ada99
I20260620 11:46:49.266687 29895 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260620 11:46:49.274338 29777 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.6.234.252" port: 44667 } attempt: 3
Jun 20 11:46:49 dist-test-slave-cjc6 krb5kdc[29621](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781956008, etypes {rep=17 tkt=17 ses=17}, kudu/127.6.234.253@KRBTEST.COM for kudu/127.6.234.252@KRBTEST.COM
I20260620 11:46:49.281772 29779 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 11:46:49.274576 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.234.252:44667 (local address 127.0.0.1:59556)
0620 11:46:49.274626 (+ 50us) negotiation.cc:107] Waiting for socket to connect
0620 11:46:49.274636 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0620 11:46:49.274672 (+ 36us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 11:46:49.274799 (+ 127us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 11:46:49.274800 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 11:46:49.274956 (+ 156us) client_negotiation.cc:189] Negotiated authn=SASL
0620 11:46:49.275000 (+ 44us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 11:46:49.275001 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 11:46:49.275837 (+ 836us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 11:46:49.275838 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 11:46:49.276624 (+ 786us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 11:46:49.276626 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 11:46:49.277069 (+ 443us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 11:46:49.277070 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 11:46:49.277118 (+ 48us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 11:46:49.277549 (+ 431us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 11:46:49.277553 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0620 11:46:49.279188 (+ 1635us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 11:46:49.280811 (+ 1623us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 11:46:49.280812 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 11:46:49.280812 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0620 11:46:49.280994 (+ 182us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 11:46:49.281194 (+ 200us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 11:46:49.281194 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 11:46:49.281195 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0620 11:46:49.281243 (+ 48us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 11:46:49.281481 (+ 238us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 11:46:49.281482 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 11:46:49.281519 (+ 37us) client_negotiation.cc:769] Sending connection context
0620 11:46:49.281571 (+ 52us) client_negotiation.cc:240] Negotiation successful
0620 11:46:49.281659 (+ 88us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":32}
I20260620 11:46:49.282080 29857 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 11:46:49.274627 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59556 (local address 127.6.234.252:44667)
0620 11:46:49.274680 (+ 53us) server_negotiation.cc:206] Beginning negotiation
0620 11:46:49.274682 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 11:46:49.274691 (+ 9us) server_negotiation.cc:407] Connection header received
0620 11:46:49.274713 (+ 22us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 11:46:49.274713 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 11:46:49.274722 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 11:46:49.274795 (+ 73us) server_negotiation.cc:226] Negotiated authn=SASL
0620 11:46:49.275072 (+ 277us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 11:46:49.275760 (+ 688us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 11:46:49.276700 (+ 940us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 11:46:49.276978 (+ 278us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 11:46:49.277026 (+ 48us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 11:46:49.279274 (+ 2248us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 11:46:49.279275 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 11:46:49.279275 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 11:46:49.279282 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0620 11:46:49.280727 (+ 1445us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 11:46:49.281086 (+ 359us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 11:46:49.281087 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 11:46:49.281087 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 11:46:49.281134 (+ 47us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 11:46:49.281309 (+ 175us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 11:46:49.281309 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 11:46:49.281310 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 11:46:49.281417 (+ 107us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 11:46:49.281463 (+ 46us) server_negotiation.cc:1035] Waiting for connection context
0620 11:46:49.281935 (+ 472us) server_negotiation.cc:299] Negotiation successful
0620 11:46:49.282001 (+ 66us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":20}
I20260620 11:46:49.284058 29777 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 02ab4448e0714124b680d996d3cd0f8a: Bootstrap starting.
I20260620 11:46:49.284973 29777 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 02ab4448e0714124b680d996d3cd0f8a: Neither blocks nor log segments found. Creating new log.
I20260620 11:46:49.285264 29777 log.cc:824] T 00000000000000000000000000000000 P 02ab4448e0714124b680d996d3cd0f8a: Log is configured to *not* fsync() on all Append() calls
I20260620 11:46:49.290431 29777 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 02ab4448e0714124b680d996d3cd0f8a: No bootstrap required, opened a new log
I20260620 11:46:49.292470 29777 raft_consensus.cc:348] T 00000000000000000000000000000000 P 02ab4448e0714124b680d996d3cd0f8a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ecbaec1ef2a8438da898ed54ae0773a2" member_type: VOTER last_known_addr { host: "127.6.234.254" port: 38981 } } peers { permanent_uuid: "02ab4448e0714124b680d996d3cd0f8a" member_type: VOTER last_known_addr { host: "127.6.234.253" port: 36053 } } peers { permanent_uuid: "258783154f684f8f8990be1bfb35a7d0" member_type: VOTER last_known_addr { host: "127.6.234.252" port: 44667 } }
I20260620 11:46:49.292604 29777 raft_consensus.cc:729] T 00000000000000000000000000000000 P 02ab4448e0714124b680d996d3cd0f8a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 02ab4448e0714124b680d996d3cd0f8a, State: Initialized, Role: FOLLOWER
I20260620 11:46:49.292680 29777 consensus_queue.cc:260] T 00000000000000000000000000000000 P 02ab4448e0714124b680d996d3cd0f8a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ecbaec1ef2a8438da898ed54ae0773a2" member_type: VOTER last_known_addr { host: "127.6.234.254" port: 38981 } } peers { permanent_uuid: "02ab4448e0714124b680d996d3cd0f8a" member_type: VOTER last_known_addr { host: "127.6.234.253" port: 36053 } } peers { permanent_uuid: "258783154f684f8f8990be1bfb35a7d0" member_type: VOTER last_known_addr { host: "127.6.234.252" port: 44667 } }
I20260620 11:46:49.293033 29777 sys_catalog.cc:564] T 00000000000000000000000000000000 P 02ab4448e0714124b680d996d3cd0f8a [sys.catalog]: configured and running, proceeding with master startup.
I20260620 11:46:49.293798 29751 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "ecbaec1ef2a8438da898ed54ae0773a2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: true dest_uuid: "02ab4448e0714124b680d996d3cd0f8a"
I20260620 11:46:49.293903 29751 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 02ab4448e0714124b680d996d3cd0f8a [term 0 FOLLOWER]: Advancing to term 1
I20260620 11:46:49.294757 29751 raft_consensus.cc:2454] T 00000000000000000000000000000000 P 02ab4448e0714124b680d996d3cd0f8a [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ecbaec1ef2a8438da898ed54ae0773a2 in term 1.
I20260620 11:46:49.295032 29750 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 02ab4448e0714124b680d996d3cd0f8a [term 1 FOLLOWER]: Refusing update from remote peer ecbaec1ef2a8438da898ed54ae0773a2: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 11:46:49.295514 29893 consensus_queue.cc:1035] T 00000000000000000000000000000000 P ecbaec1ef2a8438da898ed54ae0773a2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "02ab4448e0714124b680d996d3cd0f8a" member_type: VOTER last_known_addr { host: "127.6.234.253" port: 36053 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 11:46:49.297952 29895 catalog_manager.cc:1370] Generated new certificate authority record
I20260620 11:46:49.297959 29902 sys_catalog.cc:455] T 00000000000000000000000000000000 P 02ab4448e0714124b680d996d3cd0f8a [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "ecbaec1ef2a8438da898ed54ae0773a2" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ecbaec1ef2a8438da898ed54ae0773a2" member_type: VOTER last_known_addr { host: "127.6.234.254" port: 38981 } } peers { permanent_uuid: "02ab4448e0714124b680d996d3cd0f8a" member_type: VOTER last_known_addr { host: "127.6.234.253" port: 36053 } } peers { permanent_uuid: "258783154f684f8f8990be1bfb35a7d0" member_type: VOTER last_known_addr { host: "127.6.234.252" port: 44667 } } }
I20260620 11:46:49.298065 29902 sys_catalog.cc:458] T 00000000000000000000000000000000 P 02ab4448e0714124b680d996d3cd0f8a [sys.catalog]: This master's current role is: FOLLOWER
I20260620 11:46:49.298130 29902 sys_catalog.cc:455] T 00000000000000000000000000000000 P 02ab4448e0714124b680d996d3cd0f8a [sys.catalog]: SysCatalogTable state changed. Reason: New leader ecbaec1ef2a8438da898ed54ae0773a2. Latest consensus state: current_term: 1 leader_uuid: "ecbaec1ef2a8438da898ed54ae0773a2" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ecbaec1ef2a8438da898ed54ae0773a2" member_type: VOTER last_known_addr { host: "127.6.234.254" port: 38981 } } peers { permanent_uuid: "02ab4448e0714124b680d996d3cd0f8a" member_type: VOTER last_known_addr { host: "127.6.234.253" port: 36053 } } peers { permanent_uuid: "258783154f684f8f8990be1bfb35a7d0" member_type: VOTER last_known_addr { host: "127.6.234.252" port: 44667 } } }
I20260620 11:46:49.298178 29902 sys_catalog.cc:458] T 00000000000000000000000000000000 P 02ab4448e0714124b680d996d3cd0f8a [sys.catalog]: This master's current role is: FOLLOWER
I20260620 11:46:49.298210 29902 sys_catalog.cc:455] T 00000000000000000000000000000000 P 02ab4448e0714124b680d996d3cd0f8a [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "ecbaec1ef2a8438da898ed54ae0773a2" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ecbaec1ef2a8438da898ed54ae0773a2" member_type: VOTER last_known_addr { host: "127.6.234.254" port: 38981 } } peers { permanent_uuid: "02ab4448e0714124b680d996d3cd0f8a" member_type: VOTER last_known_addr { host: "127.6.234.253" port: 36053 } } peers { permanent_uuid: "258783154f684f8f8990be1bfb35a7d0" member_type: VOTER last_known_addr { host: "127.6.234.252" port: 44667 } } }
I20260620 11:46:49.298269 29902 sys_catalog.cc:458] T 00000000000000000000000000000000 P 02ab4448e0714124b680d996d3cd0f8a [sys.catalog]: This master's current role is: FOLLOWER
I20260620 11:46:49.299373 29895 catalog_manager.cc:1504] Loading token signing keys...
I20260620 11:46:49.302518 29913 catalog_manager.cc:1259] Loaded cluster ID: 8679e78cd6c64fd2a94ab66fe13ada99
I20260620 11:46:49.302606 29913 catalog_manager.cc:1552] T 00000000000000000000000000000000 P 02ab4448e0714124b680d996d3cd0f8a: loading cluster ID for follower catalog manager: success
W20260620 11:46:49.302772 29913 catalog_manager.cc:1576] T 00000000000000000000000000000000 P 02ab4448e0714124b680d996d3cd0f8a: acquiring CA information for follower catalog manager: Not found: root CA entry not found
W20260620 11:46:49.302858 29913 catalog_manager.cc:873] Not found: root CA entry not found: failed to prepare follower catalog manager, will retry
I20260620 11:46:49.305584 29893 sys_catalog.cc:455] T 00000000000000000000000000000000 P ecbaec1ef2a8438da898ed54ae0773a2 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "ecbaec1ef2a8438da898ed54ae0773a2" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ecbaec1ef2a8438da898ed54ae0773a2" member_type: VOTER last_known_addr { host: "127.6.234.254" port: 38981 } } peers { permanent_uuid: "02ab4448e0714124b680d996d3cd0f8a" member_type: VOTER last_known_addr { host: "127.6.234.253" port: 36053 } } peers { permanent_uuid: "258783154f684f8f8990be1bfb35a7d0" member_type: VOTER last_known_addr { host: "127.6.234.252" port: 44667 } } }
I20260620 11:46:49.306190 29893 sys_catalog.cc:458] T 00000000000000000000000000000000 P ecbaec1ef2a8438da898ed54ae0773a2 [sys.catalog]: This master's current role is: LEADER
I20260620 11:46:49.306645 29895 catalog_manager.cc:5868] T 00000000000000000000000000000000 P ecbaec1ef2a8438da898ed54ae0773a2: Generated new TSK 0
I20260620 11:46:49.306850 29895 catalog_manager.cc:1514] Initializing in-progress tserver states...
W20260620 11:46:49.332681 29781 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 11:46:49.330962 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48952 (local address 127.6.234.253:36053)
0620 11:46:49.331008 (+ 46us) server_negotiation.cc:206] Beginning negotiation
0620 11:46:49.331010 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 11:46:49.331992 (+ 982us) server_negotiation.cc:407] Connection header received
0620 11:46:49.332104 (+ 112us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 11:46:49.332105 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 11:46:49.332117 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 11:46:49.332148 (+ 31us) server_negotiation.cc:226] Negotiated authn=SASL
0620 11:46:49.332608 (+ 460us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:48952: BlockingRecv error: recv got EOF from 127.0.0.1:48952 (error 108)
Metrics: {"server-negotiator.queue_time_us":23}
W20260620 11:46:49.333480 29706 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 11:46:49.331393 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51292 (local address 127.6.234.254:38981)
0620 11:46:49.331431 (+ 38us) server_negotiation.cc:206] Beginning negotiation
0620 11:46:49.331433 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 11:46:49.332935 (+ 1502us) server_negotiation.cc:407] Connection header received
0620 11:46:49.333040 (+ 105us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 11:46:49.333041 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 11:46:49.333052 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 11:46:49.333082 (+ 30us) server_negotiation.cc:226] Negotiated authn=SASL
0620 11:46:49.333416 (+ 334us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:51292: BlockingRecv error: recv got EOF from 127.0.0.1:51292 (error 108)
Metrics: {"server-negotiator.queue_time_us":20}
W20260620 11:46:49.334264 29857 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 11:46:49.331705 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59564 (local address 127.6.234.252:44667)
0620 11:46:49.331744 (+ 39us) server_negotiation.cc:206] Beginning negotiation
0620 11:46:49.331745 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0620 11:46:49.333681 (+ 1936us) server_negotiation.cc:407] Connection header received
0620 11:46:49.333859 (+ 178us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 11:46:49.333861 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 11:46:49.333871 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 11:46:49.333902 (+ 31us) server_negotiation.cc:226] Negotiated authn=SASL
0620 11:46:49.334210 (+ 308us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:59564: BlockingRecv error: recv got EOF from 127.0.0.1:59564 (error 108)
Metrics: {"server-negotiator.queue_time_us":20}
W20260620 11:46:49.336962 29781 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 11:46:49.336025 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48954 (local address 127.6.234.253:36053)
0620 11:46:49.336068 (+ 43us) server_negotiation.cc:206] Beginning negotiation
0620 11:46:49.336070 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 11:46:49.336319 (+ 249us) server_negotiation.cc:407] Connection header received
0620 11:46:49.336433 (+ 114us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 11:46:49.336434 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 11:46:49.336453 (+ 19us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 11:46:49.336483 (+ 30us) server_negotiation.cc:226] Negotiated authn=SASL
0620 11:46:49.336918 (+ 435us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:48954: BlockingRecv error: recv got EOF from 127.0.0.1:48954 (error 108)
Metrics: {"server-negotiator.queue_time_us":18}
W20260620 11:46:49.339936 29781 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 11:46:49.339206 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48958 (local address 127.6.234.253:36053)
0620 11:46:49.339237 (+ 31us) server_negotiation.cc:206] Beginning negotiation
0620 11:46:49.339239 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 11:46:49.339471 (+ 232us) server_negotiation.cc:407] Connection header received
0620 11:46:49.339562 (+ 91us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 11:46:49.339563 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 11:46:49.339572 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 11:46:49.339599 (+ 27us) server_negotiation.cc:226] Negotiated authn=SASL
0620 11:46:49.339893 (+ 294us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:48958: BlockingRecv error: recv got EOF from 127.0.0.1:48958 (error 108)
Metrics: {"server-negotiator.queue_time_us":16}
W20260620 11:46:49.343695 29781 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 11:46:49.342944 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48960 (local address 127.6.234.253:36053)
0620 11:46:49.342977 (+ 33us) server_negotiation.cc:206] Beginning negotiation
0620 11:46:49.342979 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 11:46:49.343218 (+ 239us) server_negotiation.cc:407] Connection header received
0620 11:46:49.343314 (+ 96us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 11:46:49.343315 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 11:46:49.343325 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 11:46:49.343354 (+ 29us) server_negotiation.cc:226] Negotiated authn=SASL
0620 11:46:49.343657 (+ 303us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:48960: BlockingRecv error: recv got EOF from 127.0.0.1:48960 (error 108)
Metrics: {"server-negotiator.queue_time_us":17}
W20260620 11:46:49.349524 29781 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 11:46:49.348734 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48974 (local address 127.6.234.253:36053)
0620 11:46:49.348779 (+ 45us) server_negotiation.cc:206] Beginning negotiation
0620 11:46:49.348781 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 11:46:49.349025 (+ 244us) server_negotiation.cc:407] Connection header received
0620 11:46:49.349124 (+ 99us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 11:46:49.349125 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 11:46:49.349135 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 11:46:49.349165 (+ 30us) server_negotiation.cc:226] Negotiated authn=SASL
0620 11:46:49.349484 (+ 319us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:48974: BlockingRecv error: recv got EOF from 127.0.0.1:48974 (error 108)
Metrics: {"server-negotiator.queue_time_us":17}
W20260620 11:46:49.359634 29781 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 11:46:49.358736 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48988 (local address 127.6.234.253:36053)
0620 11:46:49.358790 (+ 54us) server_negotiation.cc:206] Beginning negotiation
0620 11:46:49.358792 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 11:46:49.359067 (+ 275us) server_negotiation.cc:407] Connection header received
0620 11:46:49.359173 (+ 106us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 11:46:49.359174 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 11:46:49.359185 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 11:46:49.359216 (+ 31us) server_negotiation.cc:226] Negotiated authn=SASL
0620 11:46:49.359581 (+ 365us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:48988: BlockingRecv error: recv got EOF from 127.0.0.1:48988 (error 108)
Metrics: {"server-negotiator.queue_time_us":31}
W20260620 11:46:49.378069 29781 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 11:46:49.377039 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48998 (local address 127.6.234.253:36053)
0620 11:46:49.377084 (+ 45us) server_negotiation.cc:206] Beginning negotiation
0620 11:46:49.377086 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 11:46:49.377378 (+ 292us) server_negotiation.cc:407] Connection header received
0620 11:46:49.377527 (+ 149us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 11:46:49.377528 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 11:46:49.377540 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 11:46:49.377572 (+ 32us) server_negotiation.cc:226] Negotiated authn=SASL
0620 11:46:49.378011 (+ 439us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:48998: BlockingRecv error: recv got EOF from 127.0.0.1:48998 (error 108)
Metrics: {"server-negotiator.queue_time_us":21}
W20260620 11:46:49.412173 29781 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 11:46:49.411367 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49014 (local address 127.6.234.253:36053)
0620 11:46:49.411416 (+ 49us) server_negotiation.cc:206] Beginning negotiation
0620 11:46:49.411418 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 11:46:49.411581 (+ 163us) server_negotiation.cc:407] Connection header received
0620 11:46:49.411653 (+ 72us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 11:46:49.411654 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 11:46:49.411668 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 11:46:49.411745 (+ 77us) server_negotiation.cc:226] Negotiated authn=SASL
0620 11:46:49.412109 (+ 364us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:49014: BlockingRecv error: recv got EOF from 127.0.0.1:49014 (error 108)
Metrics: {"server-negotiator.queue_time_us":22}
W20260620 11:46:49.478140 29781 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 11:46:49.477368 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49022 (local address 127.6.234.253:36053)
0620 11:46:49.477425 (+ 57us) server_negotiation.cc:206] Beginning negotiation
0620 11:46:49.477427 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 11:46:49.477506 (+ 79us) server_negotiation.cc:407] Connection header received
0620 11:46:49.477606 (+ 100us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 11:46:49.477608 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 11:46:49.477621 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 11:46:49.477755 (+ 134us) server_negotiation.cc:226] Negotiated authn=SASL
0620 11:46:49.478081 (+ 326us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:49022: BlockingRecv error: recv got EOF from 127.0.0.1:49022 (error 108)
Metrics: {"server-negotiator.queue_time_us":34}
W20260620 11:46:49.608062 29781 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 11:46:49.607380 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49036 (local address 127.6.234.253:36053)
0620 11:46:49.607469 (+ 89us) server_negotiation.cc:206] Beginning negotiation
0620 11:46:49.607470 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0620 11:46:49.607556 (+ 86us) server_negotiation.cc:407] Connection header received
0620 11:46:49.607628 (+ 72us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 11:46:49.607629 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 11:46:49.607639 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 11:46:49.607685 (+ 46us) server_negotiation.cc:226] Negotiated authn=SASL
0620 11:46:49.607994 (+ 309us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:49036: BlockingRecv error: recv got EOF from 127.0.0.1:49036 (error 108)
Metrics: {"server-negotiator.queue_time_us":48}
W20260620 11:46:49.865983 29781 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 11:46:49.865404 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49050 (local address 127.6.234.253:36053)
0620 11:46:49.865485 (+ 81us) server_negotiation.cc:206] Beginning negotiation
0620 11:46:49.865493 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0620 11:46:49.865505 (+ 12us) server_negotiation.cc:407] Connection header received
0620 11:46:49.865531 (+ 26us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 11:46:49.865532 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 11:46:49.865541 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 11:46:49.865606 (+ 65us) server_negotiation.cc:226] Negotiated authn=SASL
0620 11:46:49.865936 (+ 330us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:49050: BlockingRecv error: recv got EOF from 127.0.0.1:49050 (error 108)
Metrics: {"server-negotiator.queue_time_us":48}
I20260620 11:46:50.236094 29871 catalog_manager.cc:1259] Loaded cluster ID: 8679e78cd6c64fd2a94ab66fe13ada99
I20260620 11:46:50.236176 29871 catalog_manager.cc:1552] T 00000000000000000000000000000000 P 258783154f684f8f8990be1bfb35a7d0: loading cluster ID for follower catalog manager: success
I20260620 11:46:50.237093 29871 catalog_manager.cc:1574] T 00000000000000000000000000000000 P 258783154f684f8f8990be1bfb35a7d0: acquiring CA information for follower catalog manager: success
I20260620 11:46:50.237355 29871 catalog_manager.cc:1602] T 00000000000000000000000000000000 P 258783154f684f8f8990be1bfb35a7d0: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20260620 11:46:50.304020 29913 catalog_manager.cc:1574] T 00000000000000000000000000000000 P 02ab4448e0714124b680d996d3cd0f8a: acquiring CA information for follower catalog manager: success
I20260620 11:46:50.304387 29913 catalog_manager.cc:1602] T 00000000000000000000000000000000 P 02ab4448e0714124b680d996d3cd0f8a: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20260620 11:46:50.381909 29985 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 11:46:50.379161 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49052 (local address 127.6.234.253:36053)
0620 11:46:50.379287 (+ 126us) server_negotiation.cc:206] Beginning negotiation
0620 11:46:50.379289 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 11:46:50.379345 (+ 56us) server_negotiation.cc:407] Connection header received
0620 11:46:50.379420 (+ 75us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 11:46:50.379421 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 11:46:50.379473 (+ 52us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 11:46:50.379552 (+ 79us) server_negotiation.cc:226] Negotiated authn=TOKEN
0620 11:46:50.379684 (+ 132us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 11:46:50.380245 (+ 561us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 11:46:50.381072 (+ 827us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 11:46:50.381304 (+ 232us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 11:46:50.381367 (+ 63us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 11:46:50.381540 (+ 173us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0620 11:46:50.381656 (+ 116us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0620 11:46:50.381783 (+ 127us) server_negotiation.cc:1035] Waiting for connection context
0620 11:46:50.381815 (+ 32us) server_negotiation.cc:299] Negotiation successful
0620 11:46:50.381835 (+ 20us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":103,"thread_start_us":74,"threads_started":1}
I20260620 11:46:50.386040 29991 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 11:46:50.383773 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59576 (local address 127.6.234.252:44667)
0620 11:46:50.383884 (+ 111us) server_negotiation.cc:206] Beginning negotiation
0620 11:46:50.383886 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 11:46:50.383905 (+ 19us) server_negotiation.cc:407] Connection header received
0620 11:46:50.383941 (+ 36us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 11:46:50.383941 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 11:46:50.383984 (+ 43us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 11:46:50.384047 (+ 63us) server_negotiation.cc:226] Negotiated authn=TOKEN
0620 11:46:50.384128 (+ 81us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 11:46:50.384624 (+ 496us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 11:46:50.385297 (+ 673us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 11:46:50.385490 (+ 193us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 11:46:50.385543 (+ 53us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 11:46:50.385658 (+ 115us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0620 11:46:50.385821 (+ 163us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0620 11:46:50.385899 (+ 78us) server_negotiation.cc:1035] Waiting for connection context
0620 11:46:50.385924 (+ 25us) server_negotiation.cc:299] Negotiation successful
0620 11:46:50.385951 (+ 27us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":91,"thread_start_us":56,"threads_started":1}
I20260620 11:46:50.390862 29991 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 11:46:50.388181 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59590 (local address 127.6.234.252:44667)
0620 11:46:50.388219 (+ 38us) server_negotiation.cc:206] Beginning negotiation
0620 11:46:50.388221 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 11:46:50.388379 (+ 158us) server_negotiation.cc:407] Connection header received
0620 11:46:50.388416 (+ 37us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 11:46:50.388417 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 11:46:50.388429 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 11:46:50.388496 (+ 67us) server_negotiation.cc:226] Negotiated authn=TOKEN
0620 11:46:50.388610 (+ 114us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 11:46:50.389058 (+ 448us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 11:46:50.390004 (+ 946us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 11:46:50.390325 (+ 321us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 11:46:50.390395 (+ 70us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 11:46:50.390583 (+ 188us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0620 11:46:50.390614 (+ 31us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0620 11:46:50.390656 (+ 42us) server_negotiation.cc:1035] Waiting for connection context
0620 11:46:50.390764 (+ 108us) server_negotiation.cc:299] Negotiation successful
0620 11:46:50.390785 (+ 21us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":22}
I20260620 11:46:50.390964 29985 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 11:46:50.387926 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49064 (local address 127.6.234.253:36053)
0620 11:46:50.387970 (+ 44us) server_negotiation.cc:206] Beginning negotiation
0620 11:46:50.387972 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 11:46:50.388081 (+ 109us) server_negotiation.cc:407] Connection header received
0620 11:46:50.388172 (+ 91us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 11:46:50.388173 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 11:46:50.388183 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 11:46:50.388213 (+ 30us) server_negotiation.cc:226] Negotiated authn=TOKEN
0620 11:46:50.388544 (+ 331us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 11:46:50.389366 (+ 822us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 11:46:50.390279 (+ 913us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 11:46:50.390482 (+ 203us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 11:46:50.390551 (+ 69us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 11:46:50.390699 (+ 148us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0620 11:46:50.390735 (+ 36us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0620 11:46:50.390785 (+ 50us) server_negotiation.cc:1035] Waiting for connection context
0620 11:46:50.390856 (+ 71us) server_negotiation.cc:299] Negotiation successful
0620 11:46:50.390875 (+ 19us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":24}
I20260620 11:46:50.391618 29998 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 11:46:50.387966 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51304 (local address 127.6.234.254:38981)
0620 11:46:50.388114 (+ 148us) server_negotiation.cc:206] Beginning negotiation
0620 11:46:50.388116 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 11:46:50.388267 (+ 151us) server_negotiation.cc:407] Connection header received
0620 11:46:50.388314 (+ 47us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 11:46:50.388315 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 11:46:50.388362 (+ 47us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 11:46:50.388446 (+ 84us) server_negotiation.cc:226] Negotiated authn=TOKEN
0620 11:46:50.388657 (+ 211us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 11:46:50.389472 (+ 815us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 11:46:50.390578 (+ 1106us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 11:46:50.390925 (+ 347us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 11:46:50.390992 (+ 67us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 11:46:50.391331 (+ 339us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0620 11:46:50.391416 (+ 85us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0620 11:46:50.391466 (+ 50us) server_negotiation.cc:1035] Waiting for connection context
0620 11:46:50.391537 (+ 71us) server_negotiation.cc:299] Negotiation successful
0620 11:46:50.391557 (+ 20us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":123,"thread_start_us":60,"threads_started":1}
I20260620 11:46:50.393194 7083 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task1JRg7N/build/release/bin/kudu with pid 29634
I20260620 11:46:50.399485 7083 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task1JRg7N/build/release/bin/kudu with pid 29709
I20260620 11:46:50.404812 7083 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task1JRg7N/build/release/bin/kudu with pid 29784
2026-06-20T11:46:50Z chronyd exiting
[ OK ] ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys/1 (2824 ms)
[----------] 2 tests from ConnectToFollowerMasterTest (5649 ms total)
[----------] Global test environment tear-down
[==========] 56 tests from 4 test suites ran. (263644 ms total)
[ PASSED ] 55 tests.
[ FAILED ] 1 test, listed below:
[ FAILED ] SecurityITest.TestJwtMiniCluster
1 FAILED TEST